3

I have a dynamic sitemap that will change on a daily basis. Because I'm hosting on Heroku, I'm having some difficulty getting it to persist. I'm considering having the sitemap generate on request, rather than generated and saved to a static file on the server.

It's a large file with tens of thousands of entries so it takes a considerable amount of time to load. Will this affect how search engines treat my site?

| improve this question | |
  • I am assuming sitemap.xml here. There is a limit of about 50k entries for a sitemap, however, I often suggesting less- about 45k. This should still result in a rather smallish file that downloads with ease. I would not have the file generated upon request- this would slow things down even more. And I am not so sure I would be working to cache this file. How many times will this file be requested per day? Not many. Certainly not by browsers which may be a hint here. Search engines know that sitemaps can be larger. It is not a page- but a resource so speed is not an issue. – closetnoc Mar 12 '15 at 15:39
  • Cached was not the right word. It's the choice between creating the sitemap from the database on every request or serving a static file generated as a timed server job (complex on Heroku because it has an ephemeral filesystem that you can't save to unless you push a repo) – Tom Chambers Mar 12 '15 at 20:11
  • Gotcha!! I generate mine when I generate content in a batch style process. I guess if not too many people are requesting a sitemap, then it is just a matter of how fast can you create one? Can you create a sitemap fast enough to not time out? For me, that would never really work- too many pages. Systems are fast enough these days that it might work. Worth a try perhaps. You can possibly use GWT Fetch as Google or the sitemap option to experiment. – closetnoc Mar 12 '15 at 21:13
1

All browsers have a connection timeout where they give up when waiting on your server. For most browsers it is around 30 seconds. I was unable to find any documentation on what the connection timeout that Googlebot uses is, but I would expect it also to be around 30 seconds.

If you can keep your time to first byte under 30 seconds, I would expect that Googlebot would be able to fetch and use your dynamically generated sitemap.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.