5

We have a product navigation engine that allow our customers to filter products based on attributes (size, color, etc.). We are implementing an upgrade that allows them to use "multi-select" refinements. Basically they can say "show me all the red, blue, and green products". We want the search engines to crawl just the first page without any refinements. We've set the canonical of the page with refinments on it to the one without any refinements.

From a system resource standpoint I'm not sure I want google crawling all of those refinements links. The links that those refinements generate can be exponentially different based on the the order of the refinements, how many have been selected, etc. It just seems like a waste for google to try and index those pages.

Should I put a no-follow tag on those refinement URL's so that google doesn't even crawl them?

Improve this question

3 Answers 3

Reset to default
5

Nofollow doesn't stop Google from crawling pages, it stops any 'link juice' being passed to those pages.

Robots.txt is what you want to prevent google from crawling pages.

http://www.robotstxt.org/robotstxt.html

Improve this answer
3
  • Apparently I don't undertsand then why google says here: google.com/support/webmasters/bin/answer.py?answer=96569 "In general, we don't follow them. This means that Google does not transfer PageRank or anchor text across these links. Essentially, using nofollow causes us to drop the target links from our overall graph of the web." or "Crawl prioritization: Search engine robots can't sign in or register, so there's no reason to invite Googlebot to follow ... Using nofollow on these links enables Googlebot to crawl other pages you'd prefer to see in Google's index. "
    – Paul Lemke
    Jun 20, 2011 at 12:19
  • That's what I'm saying, bar "Essentially, using nofollow causes us to drop the target links from our overall graph of the web."... Feel free to correct me if anyone knows better, but using nofollow could never harm the target link, it's a "neutral vote" as such...
    – Anonymous
    Jun 20, 2011 at 13:07
  • Robots.txt is not appropriate here as these pages can still appear in the index but do not pass on link authority. The "follow, noindex" is the best option here as per Joshak's suggestion. This outlines it seomoz.org/learn-seo/robotstxt
    – Sandy Lee
    Feb 29, 2012 at 14:46
1

If you're sure Google is reaching the all of your products, and it seems like you are since you're using canonical URLs, then it's a good idea to prevent Google from crawling those pages since all it will do is eat up your bandwidth and server resources.

Using nofollow on any internal links is a good start. I'd also consider using the robots meta tag or x-robots-tag HTTP header to tell them not to index it, too. That way you're sure even if they find those URLs through external linking or otherwise that they will definitely not index those page. Blocking them via robots.txt is also a good thing to since you can use wildcards to catch the pages with query strings, etc.

Improve this answer
1
  • Don't use nofollow as it will prevent link authority being passed but the page can still be crawled =)
    – Sandy Lee
    Feb 29, 2012 at 14:48
1

Since you mentioned the pages can be "exponentially different" you might want to consider using the "follow, noindex" meta tag, the different classifiers can provide Google with additional information about the product so it is potentially beneficial to let them know about the additional content even if you don't want it indexed (this can help for long tail queries).

Improve this answer
1
  • This is the best approach =)
    – Sandy Lee
    Feb 29, 2012 at 14:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.