Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

we rewrote the sitemap creation process. Now the sitemap contains 43.000 URLs. 20k more than before. We have daily changing in URLs. The script that is creating the complete sitemap takes more than 30h. So we can not build it every day. Lets say that increasing the speed of the script is not possible.

What should I do? A: Stay with the 23k URLs and update it daily B: Increase number of URLs to 43k and update it weekly

share|improve this question
1  
Have you investigated why generating the sitemap takes over a full day? That seems ridiculous. (I'm assuming your site data–entries, pages, etc–is in a database of some kind.) Something is being horribly inefficient here. –  Su' Sep 14 '12 at 17:22
1  
@Su' agreed, I've seen database-driven sitemaps for 10x that many URLs created in less than 5 minutes. However, if fixing the script is literally "not possible," we need to know how important the 20k new URLs are compared to the first 23K, and how many URLs change per day. Then you can decide on average "how wrong" your sitemaps will be under either scenario. –  joshuahedlund Sep 14 '12 at 21:00
1  
@joshuahedlund True. I just don't think blindly accepting the script as a fixed point is viable, other than as some kind of thought experiment. The real problem here is the inefficiency, and such a restriction on responses at least has to be explained. Even if this script were directly trawling the filesystem, I can't see how it would take 30 hours to run. That just doesn't feel like a limitation; it seems more likely bad programming. –  Su' Sep 14 '12 at 22:45
    
If your developer or agency can't fix this, you should hire someone else. I've generated sitemaps with much urls, this should only take 2 minutes, no more. –  Baptiste Placé Sep 17 '12 at 10:07
    
Many here seem to assume the script has access to the database. On the other hand, if it has to actually crawl all pages, it may purposely default to throttle down amount of pages/second downloaded/analyzed to prevent killing the webserver :) –  Tom Sep 18 '12 at 0:50
add comment

1 Answer

What you must do is split the sitemap, using the siteindex syntax. You probably have different sections/silos on the site, e.g. blog / forum / e-commerce. Split them by sections/silos, then you may even split them again by category. Make the different sitemaps respect a semantic rule.

Google will love it, and you will love it too in WMT. More info on this : http://www.seomoz.org/blog/multiple-xml-sitemaps-increased-indexation-and-traffic

share|improve this answer
    
While there will be a need for a sitemap index soon just based on the number of URLs, the actual problem seems to be that it takes a huge amount of time to generate. It's unclear why generating multiples is going to address that. –  Su' Sep 14 '12 at 17:27
    
Under all circumstances, a solution could still be to split the site-scan into multiple smaller projects, updating e.g. one sitemap file every day. –  Tom Sep 18 '12 at 0:52
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.