Say i have the following route http://<my-domain>/{category}/subjects/{id}/{title}

the ones in the brackets are dynamic, I'm struggling with what is better or any better way to let google crawl through all these dynamic links

Approach 1: manually doing the job by removing or adding the record to site map and updating <lastmod>

Approach 2: create a page that includes all those links and reference that page in sitemap.xml

The second approach can be generated as a plain html file which is generated from the server app. Or, a simple webform aspx page that dynamically generates those links without having to create an html file.

1 Answer 1


You should create an XML sitemap and dynamically add new dynamic URLs to it. There is no need to use lastmod in an XML sitemap. Google says they ignore lastmod because few sites keep it up to date. Googlebot will notice any new URLs in the XML sitemap and will come crawl them whether on to they have a lastmod specified.

You should not create a single page that links to all your dynamic URLs. HTML sitemaps don't work well for SEO anymore. XML sitemaps work fine for getting your content crawled.

When you have an XML sitemap Googlebot will come crawl all your content. However, it is likely that most of the URLs in it won't get indexed. If any of them do get indexed, they won't rank well. This is called The Sitemap Paradox. To get most of your dynamic URLs indexed and ranking, they need to link to each other. That is why this site has the "Related Questions" section on each question page.

Your home page should link to a few of your best and most recent URLs. Those URLs will in turn pass some of that link juice on to other dynamic URLs.

Once some of your URLs get external links, the link juice from external links will be passed further to other dynamic URLs as well.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.