Extract URLs from all Child XML sitemaps found in Sitemap Index

Question

I have a main XML Sitemap Index https://www.example.com/sitemap-index.xml that has multiple child XML sitemaps underneath it:

• https://www.example.com/sitemap1.xml
• https://www.example.com/sitemap2.xml
• ...
• https://www.example.com/sitemap10.xml

Is there a way to extract ALL the URLs found across ALL the child XML sitemaps using curl or wget from a Linux terminal command?

I was able to extract the URLs from a particular child XML sitemap using curl:

curl https://www.example.com/sitemap1.xml | grep -e loc | sed 's|<loc>\(.*\)<\/loc>$|\1|g' > sitemap1.txt

but I am trying to do it for all 10 XML sitemaps instead of manually typing this command and changing manually the XML sitemap path.

In some cases there is no pattern to the child XML sitemap path, so if I could somehow use the curl command that applies to any child XML sitemaps found in the main one, that'd be amazing!

Answer 1

You just need to use that same command on the sitemap index, but instead of outputting all the URLs, loop over them. You can accomplish by sticking it into backticks and looping over it with a for loop:

for sitemap in `curl https://www.example.com/sitemap-index.xml | grep -e loc | sed 's|<loc>\(.*\)<\/loc>$|\1|g'`; do echo "$sitemap"; done

Then you add the command to process each sitemap:

for sitemap in `curl https://www.example.com/sitemap-index.xml | grep -e loc | sed 's|<loc>\(.*\)<\/loc>$|\1|g'`; do curl "$sitemap" | grep -e loc | sed 's|<loc>\(.*\)<\/loc>$|\1|g' ; done

Finally you put the entire thing in parenthesis (with spaces inside) so that the entire command can be piped to a text file:

( for sitemap in `curl https://www.example.com/sitemap-index.xml | grep -e loc | sed 's|<loc>\(.*\)<\/loc>$|\1|g'`; do curl "$sitemap" | grep -e loc | sed 's|<loc>\(.*\)<\/loc>$|\1|g' ; done ) > sitemap.txt

The only other things that I would suggest:

• Use curl -s so that curl doesn't output info about what it is connecting to, just the contents of the URLs
• Use grep -oiE '<loc>.*?</loc>' to pull out just the contents of the <loc> tags in a case insensitive manner so that your command works even if a sitemap file doesn't have new lines or has upper case tags.
• There is a shorter sed command to remove the loc tags (and is case insensitive): sed -E 's|<\/?loc>||gi'

( for sitemap in `curl -s https://www.example.com/sitemap-index.xml | grep -oiE '<loc>.*?</loc>' | sed -E 's|<\/?loc>||gi'`; do curl -s "$sitemap" | grep -oiE '<loc>.*?</loc>' | sed -E 's|<\/?loc>||gi' ; done ) > sitemap.txt

It is also possible to eliminate the need for sed altogether by using negative lookahead patterns in grep so that it only outputs the URL without the tags around it: grep -oiP '(?<=<loc>)(.*?)(?=</loc>)'

( for sitemap in `curl -s https://www.example.com/sitemap-index.xml | grep -oiP '(?<=<loc>)(.*?)(?=</loc>)'`; do curl -s "$sitemap" | grep -oiP '(?<=<loc>)(.*?)(?=</loc>)' ; done ) > sitemap.txt

I tested this against my WordPress blog. When I input the sitemap index, it outputs a single sitemap text file with all the URLs in all the small sitemaps created by Yoast.