I am currently working on a shopping / auction website. The involving party who's doing SEO consultancy is insisting to generate URLs for every single combination of products listed on the website, examples below. It just doesn't feel right to me being that each product / category page already has a canonical links so I am really now sure how much gain the new sitemap would make. Would someone comment on the decision and give suggestions / recommendations in the given context? Thanks.


  • www.mywebsite.com/CPU
  • www.mywebsite.com/CPU/Intel
  • www.mywebsite.com/CPU/Intel/Core2Duo
  • www.mywebsite.com/CPU/AMD
  • www.mywebsite.com/CPU/AMD/Phenom
  • www.mywebsite.com/RAM
  • etc...
  • You mention that each page already has a canonical - what does the canonical look like? The SEO guy's suggestions seem to make perfect sense to me. – Eric Petroelje Nov 24 '10 at 16:02

There isn't enough information in your question to provide a "perfect" answer. I tend to think that you might be barking up the wrong tree, but I do so by assuming a few things as follows.

I assume:

  • you have so many (single) product pages now, Google hasn't crawled them all (more than 100,000?)
  • products will come and go with such frequency that spidering is a never-ending concern, and of critical importance
  • without a sitemap, you probably won't get a crawl of all product pages so you rely on the sitemap to expose all pages you need crawled

My suggestions:

  • architect a browseable hierarchy for the benefit of the spiders (only); it's clickable by users, of course, but it's purpose is just for the spider to crawl
  • the browse hierarchy needs every product to be no more than four pages deep
  • construct your URL rewrite to follow the crawl (i.e. first directory = product-catalogue (or similar such phrase), second directory = drill down to first level of product specificity (i.e. "storage"), last directory = a keyword, a hyphen and product ID, last bit of URL string is Product-Name.html)
  • design-wise, show the browseable directory links insignificantly to users; you really made this for the spider and from a usability standpoint, it's better if your users click through the existing UI
  • steer the spider by designating all other internal links as NoFollow (some may say not all other internal links; whatever the follow/nofollow approach, you need to encourage the spider to travel the easy hierarchical path, and not bounce around the site using all the other natural links)

In summary, what I'm saying is that you should not make a page for every combination of products, afterall, that's an infinite number and an impossible task. Respectfully, I disagree with your SEO guy and @Eric. Instead, I'd be certain to architect a browse-able "catalogue" for the benefit of the spiders, and marry your URL rewrite logic with the drill down clicks AND also your anchor text as the spider drills down. I'd be happy to share with you a URL to serve as an example outside of this forum (email: chris@adragna.com).

If you're still considering the combo pages, the existing URL logic you wrote and the site maps, work with this math: you can have up to 50,000 URLs per map and up to 1000 site maps. That a maximum of 50 million pages... if you want to work backwards from that you can use it to figure out the upper limits of how many product combinations to spin out.

  • Your assumptions are absolutely correct. Apparently Amazon has 20 millions of URLs in their sitemap. – Jeff Nov 25 '10 at 12:03
  • I couldn't think of a better way to implement browseable hierarchy just for the crawler. In the case of Amazon, do they provide a full list of products per category? – Jeff Nov 25 '10 at 12:06

I think only canonical versions should be on sitemap. Follow:

  • www.mywebsite.com/CPU
  • www.mywebsite.com/CPU/Intel
  • www.mywebsite.com/CPU/Intel/Core2Duo
  • www.mywebsite.com/CPU/AMD
  • www.mywebsite.com/CPU/AMD/Phenom

If all those links just point to www.mywebsite.com/CPU, then only add this url to sitemap. If each page is different (even if it is just the title), then you need to add them all to sitemap, but beware of duplicating content.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.