123 reputation
3
bio website
location
age
visits member for 3 years, 9 months
seen Jul 22 '12 at 2:40

Nov
25
comment sitemap that contains millions of links/URLs
I couldn't think of a better way to implement browseable hierarchy just for the crawler. In the case of Amazon, do they provide a full list of products per category?
Nov
25
comment sitemap that contains millions of links/URLs
Your assumptions are absolutely correct. Apparently Amazon has 20 millions of URLs in their sitemap.