1

I have a main XML Sitemap Index https://www.example.com/sitemap-index.xml that has multiple child XML sitemaps underneath it:

  • https://www.example.com/sitemap1.xml
  • https://www.example.com/sitemap2.xml
  • ...
  • https://www.example.com/sitemap10.xml

Is there a way to extract ALL the URLs found across ALL the child XML sitemaps using curl or wget from a Linux terminal command?

I was able to extract the URLs from a particular child XML sitemap using curl:

curl https://www.example.com/sitemap1.xml | grep -e loc | sed 's|<loc>\(.*\)<\/loc>$|\1|g' > sitemap1.txt

but I am trying to do it for all 10 XML sitemaps instead of manually typing this command and changing manually the XML sitemap path.

In some cases there is no pattern to the child XML sitemap path, so if I could somehow use the curl command that applies to any child XML sitemaps found in the main one, that'd be amazing!

1 Answer 1

2

You just need to use that same command on the sitemap index, but instead of outputting all the URLs, loop over them. You can accomplish by sticking it into backticks and looping over it with a for loop:

for sitemap in `curl https://www.example.com/sitemap-index.xml | grep -e loc | sed 's|<loc>\(.*\)<\/loc>$|\1|g'`; do echo "$sitemap"; done

Then you add the command to process each sitemap:

for sitemap in `curl https://www.example.com/sitemap-index.xml | grep -e loc | sed 's|<loc>\(.*\)<\/loc>$|\1|g'`; do curl "$sitemap" | grep -e loc | sed 's|<loc>\(.*\)<\/loc>$|\1|g' ; done

Finally you put the entire thing in parenthesis (with spaces inside) so that the entire command can be piped to a text file:

( for sitemap in `curl https://www.example.com/sitemap-index.xml | grep -e loc | sed 's|<loc>\(.*\)<\/loc>$|\1|g'`; do curl "$sitemap" | grep -e loc | sed 's|<loc>\(.*\)<\/loc>$|\1|g' ; done ) > sitemap.txt

The only other things that I would suggest:

  • Use curl -s so that curl doesn't output info about what it is connecting to, just the contents of the URLs
  • Use grep -oiE '<loc>.*?</loc>' to pull out just the contents of the <loc> tags in a case insensitive manner so that your command works even if a sitemap file doesn't have new lines or has upper case tags.
  • There is a shorter sed command to remove the loc tags (and is case insensitive): sed -E 's|<\/?loc>||gi'
( for sitemap in `curl -s https://www.example.com/sitemap-index.xml | grep -oiE '<loc>.*?</loc>' | sed -E 's|<\/?loc>||gi'`; do curl -s "$sitemap" | grep -oiE '<loc>.*?</loc>' | sed -E 's|<\/?loc>||gi' ; done ) > sitemap.txt

It is also possible to eliminate the need for sed altogether by using negative lookahead patterns in grep so that it only outputs the URL without the tags around it: grep -oiP '(?<=<loc>)(.*?)(?=</loc>)'

( for sitemap in `curl -s https://www.example.com/sitemap-index.xml | grep -oiP '(?<=<loc>)(.*?)(?=</loc>)'`; do curl -s "$sitemap" | grep -oiP '(?<=<loc>)(.*?)(?=</loc>)' ; done ) > sitemap.txt

I tested this against my WordPress blog. When I input the sitemap index, it outputs a single sitemap text file with all the URLs in all the small sitemaps created by Yoast.

3
  • thanks a lot, it worked like a charm! I used this version ( for sitemap in curl -s example.com/sitemap-index.xml | grep -oiE '<loc>.*?</loc>' | sed -E 's|<\/?loc>||gi'; do curl -s "$sitemap" | grep -oiE '<loc>.*?</loc>' | sed -E 's|<\/?loc>||gi' ; done ) > sitemap.txt
    – Lernisios
    Feb 18, 2022 at 3:30
  • and this one ( for sitemap in curl -s example.com/sitemap-index.xml | grep -oiP '(?<=<loc>)(.*?)(?=</loc>)'; do curl -s "$sitemap" | grep -oiP '(?<=<loc>)(.*?)(?=</loc>)' ; done ) > sitemap.txt as they've cleaned up the <loc> from the sitemaps and it had some images URLs too like this <image:loc> ; both commands worked great and did the same thing, I prefer the last command as it seems the cleanest
    – Lernisios
    Feb 18, 2022 at 3:30
  • You don’t need to create a subshell with parentheses to send the output to a file: for sitemap in ...; do ...; done > sitemap.txt.
    – bfontaine
    Oct 25 at 9:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.