1

I have a gz index sitemap file that lists 4 different sitemaps that each are 50mb big. What I want to do is I want to get all the URLs in those 4 sitemap files.

I tried to use https://robhammond.co/tools/xml-extract but it couldn't deal with the index file properly. My main sitemap.xml.gz only lists the other xml files so I end up with a locally stored files that I download and not a list of URLs from them.

How can I either:

  1. Use the above tool with all four sitemaps
  2. Extract the URLs from the downloaded sitemap files
New contributor
Christoph Schachner is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
1

I prefer to use command line tools to extract sitemap URLs. Most sitemaps have each URL on its own line which make them work really nicely with Unix command line tools. I can easily extract your four sitemap URLs from your index sitemap:

$ curl -s https://www.example.com/sitemap_index.xml.gz | gunzip | grep -oE 'https://[^<]+'
https://www.example.com/sitemap1.xml.gz
https://www.example.com/sitemap2.xml.gz
https://www.example.com/sitemap3.xml.gz
https://www.example.com/sitemap4.xml.gz

You can either paste each of those four URLs into a tool similar to the one you listed, or you can use command line tools to examine them further:

$ curl -s https://www.example.com/sitemap1.xml.gz | gunzip | grep '<loc' | grep -oE 'https://[^<]+'
https://www.example.com/en/c1_Bags
https://www.example.com/de/c1_Taschen
https://www.example.com/fr/c1_Sacs
....

You can also sitemaps with any text editor. You might have to decomress them with a unzip program first. (That is what gunzip does in my command line examples above.)

  • Thanks for your answer. I already have my 4 sitemaps (sitemap1.xml) downloaded. Which tool do I need to put in the code you send above? – Christoph Schachner Jan 11 at 15:10
  • @ChristophSchachner Those are shell commands - they are meant to be run at a Linux command-line prompt or in a similar unix-like environment. – Maximillian Laumeister Jan 11 at 16:47
  • Or just open the files in a text editor after unzipping them. – Stephen Ostermiller Jan 11 at 17:17
  • @StephenOstermiller when I unzip them and open them with a text editor, how do I extract only the URLs? – Christoph Schachner Jan 11 at 18:19

Your Answer

Christoph Schachner is a new contributor. Be nice, and check out our Code of Conduct.

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.