Reputation
Top tag
Next privilege 125 Rep.
Vote down
Badges
3
Newest
 Scholar
Impact
~462 people reached

Nov
25
comment sitemap that contains millions of links/URLs
I couldn't think of a better way to implement browseable hierarchy just for the crawler. In the case of Amazon, do they provide a full list of products per category?
Nov
25
comment sitemap that contains millions of links/URLs
Your assumptions are absolutely correct. Apparently Amazon has 20 millions of URLs in their sitemap.