How do I get a list of all the URLs from multiple sitemaps which are listed using an index sitemap?

Question

I have a gz index sitemap file that lists 4 different sitemaps that each are 50mb big. What I want to do is I want to get all the URLs in those 4 sitemap files.

I tried to use https://robhammond.co/tools/xml-extract but it couldn't deal with the index file properly. My main sitemap.xml.gz only lists the other xml files so I end up with a locally stored files that I download and not a list of URLs from them.

How can I either:

1. Use the above tool with all four sitemaps
2. Extract the URLs from the downloaded sitemap files

Answer 1

I prefer to use command line tools to extract sitemap URLs. Most sitemaps have each URL on its own line which make them work really nicely with Unix command line tools. I can easily extract your four sitemap URLs from your index sitemap:

$ curl -s https://www.example.com/sitemap_index.xml.gz | gunzip | grep -oE 'https://[^<]+'
https://www.example.com/sitemap1.xml.gz
https://www.example.com/sitemap2.xml.gz
https://www.example.com/sitemap3.xml.gz
https://www.example.com/sitemap4.xml.gz

You can either paste each of those four URLs into a tool similar to the one you listed, or you can use command line tools to examine them further:

$ curl -s https://www.example.com/sitemap1.xml.gz | gunzip | grep '<loc' | grep -oE 'https://[^<]+'
https://www.example.com/en/c1_Bags
https://www.example.com/de/c1_Taschen
https://www.example.com/fr/c1_Sacs
....

You can also sitemaps with any text editor. You might have to decomress them with a unzip program first. (That is what gunzip does in my command line examples above.)

Answer 2

If the sitemaps aren't gzipped and you don't mind running PHP, here's a script you can use:

<?php

$index = file_get_contents( $argv[1] );
$log = 'sitemap.log';
preg_match_all( '#<loc>([^<]+)</loc>#', $index, $matches );
foreach ( $matches[1] as $match ) {
    $sitemap = file_get_contents( $match );
    file_put_contents( $log, $match . PHP_EOL, FILE_APPEND );
    preg_match_all( '#<loc>([^<]+)</loc>#', $sitemap, $url_matches );
    foreach ( $url_matches[1] as $url_match ) {
        file_put_contents( $log, ' - ' . $url_match . PHP_EOL, FILE_APPEND );
    }
}

Run the script with php download-sitemap-urls.php <sitemap-index-url>, where:

• download-sitemap-urls.php is a file containing the script above.
• <sitemap-index-url> is the URL of the sitemap index.

The URLs will be written to a sitemap.log file in the same directory as the PHP script.