Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am wondering what the best approach would be to generating a sitemap for google to crawl with a highly dynamic url structure.

We have a webapp that assigns custom uris to all the users in our database for direct access to their content. All the uris are namespaced to one endpoint. The uris are chosen by the user who owns the profile and assume a human-readable form ('/profile/someguy')

Would google require that all possible uris be documented in the sitemap? Wouldn't this lead to a ridiculously large xml file? It would be possible to periodically generate this using a script and break it up into smaller sitemaps if that is what's necessary.

I have searched the webmaster docs for answers to this, but they largely only cover dynamic urls that use query strings, not ones that are generated in a readable format.

Any guidance is much appreciated!

share|improve this question

migrated from serverfault.com Mar 8 '12 at 2:30

This question came from our site for system and network administrators.

up vote 1 down vote accepted

Include every URL you want to notify Google about in your sitemap. Sitemaps can be very large in size (50,000 URLs) but if that isn't enough you can sitemap index files to link multiple sitemaps together allowing for an unlimited amount of URLs to be included in your sitemaps.

share|improve this answer

You should have a script coded that will query your database and re-generate your XML sitemap file say every few days. Set a cronjob to execute the script, and have it re-submitted to Google and Bing using their API. This way it's always up to date, and the search engines are always getting a fresh copy.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.