we rewrote the sitemap creation process. Now the sitemap contains 43.000 URLs. 20k more than before. We have daily changing in URLs. The script that is creating the complete sitemap takes more than 30h. So we can not build it every day. Lets say that increasing the speed of the script is not possible.

What should I do? A: Stay with the 23k URLs and update it daily B: Increase number of URLs to 43k and update it weekly

  • 1
    Have you investigated why generating the sitemap takes over a full day? That seems ridiculous. (I'm assuming your site data–entries, pages, etc–is in a database of some kind.) Something is being horribly inefficient here.
    – Su'
    Sep 14, 2012 at 17:22
  • 1
    @Su' agreed, I've seen database-driven sitemaps for 10x that many URLs created in less than 5 minutes. However, if fixing the script is literally "not possible," we need to know how important the 20k new URLs are compared to the first 23K, and how many URLs change per day. Then you can decide on average "how wrong" your sitemaps will be under either scenario. Sep 14, 2012 at 21:00
  • 1
    @joshuahedlund True. I just don't think blindly accepting the script as a fixed point is viable, other than as some kind of thought experiment. The real problem here is the inefficiency, and such a restriction on responses at least has to be explained. Even if this script were directly trawling the filesystem, I can't see how it would take 30 hours to run. That just doesn't feel like a limitation; it seems more likely bad programming.
    – Su'
    Sep 14, 2012 at 22:45
  • If your developer or agency can't fix this, you should hire someone else. I've generated sitemaps with much urls, this should only take 2 minutes, no more. Sep 17, 2012 at 10:07
  • Many here seem to assume the script has access to the database. On the other hand, if it has to actually crawl all pages, it may purposely default to throttle down amount of pages/second downloaded/analyzed to prevent killing the webserver :)
    – Tom
    Sep 18, 2012 at 0:50

1 Answer 1


What you must do is split the sitemap, using the siteindex syntax. You probably have different sections/silos on the site, e.g. blog / forum / e-commerce. Split them by sections/silos, then you may even split them again by category. Make the different sitemaps respect a semantic rule.

Google will love it, and you will love it too in WMT. More info on this : http://www.seomoz.org/blog/multiple-xml-sitemaps-increased-indexation-and-traffic

  • While there will be a need for a sitemap index soon just based on the number of URLs, the actual problem seems to be that it takes a huge amount of time to generate. It's unclear why generating multiples is going to address that.
    – Su'
    Sep 14, 2012 at 17:27
  • Under all circumstances, a solution could still be to split the site-scan into multiple smaller projects, updating e.g. one sitemap file every day.
    – Tom
    Sep 18, 2012 at 0:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.