You just need to use that same command on the sitemap index, but instead of outputting all the URLs, loop over them. You can accomplish by sticking it into backticks and looping over it with a for loop:
for sitemap in `curl https://www.example.com/sitemap-index.xml | grep -e loc | sed 's|<loc>\(.*\)<\/loc>$|\1|g'`; do echo "$sitemap"; done
Then you add the command to process each sitemap:
for sitemap in `curl https://www.example.com/sitemap-index.xml | grep -e loc | sed 's|<loc>\(.*\)<\/loc>$|\1|g'`; do curl "$sitemap" | grep -e loc | sed 's|<loc>\(.*\)<\/loc>$|\1|g' ; done
Finally you put the entire thing in parenthesis (with spaces inside) so that the entire command can be piped to a text file:
( for sitemap in `curl https://www.example.com/sitemap-index.xml | grep -e loc | sed 's|<loc>\(.*\)<\/loc>$|\1|g'`; do curl "$sitemap" | grep -e loc | sed 's|<loc>\(.*\)<\/loc>$|\1|g' ; done ) > sitemap.txt
The only other things that I would suggest:
- Use
curl -s
so that curl doesn't output info about what it is connecting to, just the contents of the URLs
- Use
grep -oiE '<loc>.*?</loc>'
to pull out just the contents of the <loc>
tags in a case insensitive manner so that your command works even if a sitemap file doesn't have new lines or has upper case tags.
- There is a shorter sed command to remove the loc tags (and is case insensitive):
sed -E 's|<\/?loc>||gi'
( for sitemap in `curl -s https://www.example.com/sitemap-index.xml | grep -oiE '<loc>.*?</loc>' | sed -E 's|<\/?loc>||gi'`; do curl -s "$sitemap" | grep -oiE '<loc>.*?</loc>' | sed -E 's|<\/?loc>||gi' ; done ) > sitemap.txt
It is also possible to eliminate the need for sed
altogether by using negative lookahead patterns in grep so that it only outputs the URL without the tags around it: grep -oiP '(?<=<loc>)(.*?)(?=</loc>)'
( for sitemap in `curl -s https://www.example.com/sitemap-index.xml | grep -oiP '(?<=<loc>)(.*?)(?=</loc>)'`; do curl -s "$sitemap" | grep -oiP '(?<=<loc>)(.*?)(?=</loc>)' ; done ) > sitemap.txt
I tested this against my WordPress blog. When I input the sitemap index, it outputs a single sitemap text file with all the URLs in all the small sitemaps created by Yoast.