Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I currently am tasked with submitting our website sitemap to the search engines every week.

We have a module which does offer sitemap generation but we find using it does not work very well as not all pages are included and it does not split the sitemap by content.

I've used various (online and offline) tools to generate the sitemaps which is not the problem. The problem is that after every generation (which takes most of each Monday) I have to manually go through the sitemap and categorise the links in to products, pages, categories and sub categories.

I've experimented successfully with XSL to split the sitemap but it is still a labour intensive process.

Does anyone know of a good method to split the sitemap?

Currently there are around 20,000 links (iirc) in total.

share|improve this question
Why do they need to be in different sitemaps? Each sitemap file can be 50MB and hold 50,000 URLs. The URLs will get submitted to search engines just fine in a single sitemap. –  Stephen Ostermiller Jan 6 at 16:35
Are you trying to split the sitemap based on a URL pattern? If so, please provide some examples. –  Stephen Ostermiller Jan 6 at 16:36
When your sitemap is generated, is each URL on a single line by itself? If so, it would be pretty trivial to split the file with a small Perl script. If entries span multiple lines or there are multiple entries per line, then scripting languages may be of slightly less value. –  Stephen Ostermiller Jan 6 at 16:38
It is something which our SEO guys have forwarded to me to do. Apparently it is advantageous for indexing speed and will be helpful if something goes wrong in pinpointing the problem. A pattern could be used for some, for example weburl.co.uk/category/subcategory/product but this would not work as across the board. Also the sitemaps I have generated up to now have all been formatted so each URL is on multiple lines. –  James Jan 6 at 17:01
If a pattern won't work for all the URLs, what criteria do you use to split the file when you do it by hand? –  Stephen Ostermiller Jan 6 at 18:16

1 Answer 1

There are not off the shelf products that are going to do what you want. You could write a script to do it. It wouldn't be too hard in a language like Perl or Python.

If you sitemap entries were on single lines, it would be much easier. You will need to hook up an XML parser instead of just reading the file line by line.

Once you can read the file line by line in the script, you place them in various buckets using the same criteria that you use to do so by hand.

share|improve this answer
Thanks Stephen. I have no experience with either of those languages but I'm sure I can figure something out, I'll definitely look in to it. I would +1 your answer but my rep is too low, I'll give others a chance to answer before marking it as 'the' answer also. :) –  James Jan 10 at 16:39

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.