I have a gz index sitemap file that lists 4 different sitemaps that each are 50mb big. What I want to do is I want to get all the URLs in those 4 sitemap files.

I tried to use https://robhammond.co/tools/xml-extract but it couldn't deal with the index file properly. My main sitemap.xml.gz only lists the other xml files so I end up with a locally stored files that I download and not a list of URLs from them.

How can I either:

  1. Use the above tool with all four sitemaps
  2. Extract the URLs from the downloaded sitemap files

2 Answers 2


I prefer to use command line tools to extract sitemap URLs. Most sitemaps have each URL on its own line which make them work really nicely with Unix command line tools. I can easily extract your four sitemap URLs from your index sitemap:

$ curl -s https://www.example.com/sitemap_index.xml.gz | gunzip | grep -oE 'https://[^<]+'

You can either paste each of those four URLs into a tool similar to the one you listed, or you can use command line tools to examine them further:

$ curl -s https://www.example.com/sitemap1.xml.gz | gunzip | grep '<loc' | grep -oE 'https://[^<]+'

You can also sitemaps with any text editor. You might have to decomress them with a unzip program first. (That is what gunzip does in my command line examples above.)

  • Thanks for your answer. I already have my 4 sitemaps (sitemap1.xml) downloaded. Which tool do I need to put in the code you send above? Jan 11, 2019 at 15:10
  • @ChristophSchachner Those are shell commands - they are meant to be run at a Linux command-line prompt or in a similar unix-like environment. Jan 11, 2019 at 16:47
  • Or just open the files in a text editor after unzipping them. Jan 11, 2019 at 17:17
  • @StephenOstermiller when I unzip them and open them with a text editor, how do I extract only the URLs? Jan 11, 2019 at 18:19

If the sitemaps aren't gzipped and you don't mind running PHP, here's a script you can use:


$index = file_get_contents( $argv[1] );
$log = 'sitemap.log';
preg_match_all( '#<loc>([^<]+)</loc>#', $index, $matches );
foreach ( $matches[1] as $match ) {
    $sitemap = file_get_contents( $match );
    file_put_contents( $log, $match . PHP_EOL, FILE_APPEND );
    preg_match_all( '#<loc>([^<]+)</loc>#', $sitemap, $url_matches );
    foreach ( $url_matches[1] as $url_match ) {
        file_put_contents( $log, ' - ' . $url_match . PHP_EOL, FILE_APPEND );

Run the script with php download-sitemap-urls.php <sitemap-index-url>, where:

  • download-sitemap-urls.php is a file containing the script above.
  • <sitemap-index-url> is the URL of the sitemap index.

The URLs will be written to a sitemap.log file in the same directory as the PHP script.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.