Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

We have a product navigation engine that allow our customers to filter products based on attributes (size, color, etc.). We are implementing an upgrade that allows them to use "multi-select" refinements. Basically they can say "show me all the red, blue, and green products". We want the search engines to crawl just the first page without any refinements. We've set the canonical of the page with refinments on it to the one without any refinements.

From a system resource standpoint I'm not sure I want google crawling all of those refinements links. The links that those refinements generate can be exponentially different based on the the order of the refinements, how many have been selected, etc. It just seems like a waste for google to try and index those pages.

Should I put a no-follow tag on those refinement URL's so that google doesn't even crawl them?

share|improve this question
add comment

3 Answers

Nofollow doesn't stop Google from crawling pages, it stops any 'link juice' being passed to those pages.

Robots.txt is what you want to prevent google from crawling pages.

http://www.robotstxt.org/robotstxt.html

share|improve this answer
    
Apparently I don't undertsand then why google says here: google.com/support/webmasters/bin/answer.py?answer=96569 "In general, we don't follow them. This means that Google does not transfer PageRank or anchor text across these links. Essentially, using nofollow causes us to drop the target links from our overall graph of the web." or "Crawl prioritization: Search engine robots can't sign in or register, so there's no reason to invite Googlebot to follow ... Using nofollow on these links enables Googlebot to crawl other pages you'd prefer to see in Google's index. " –  Paul Lemke Jun 20 '11 at 12:19
    
That's what I'm saying, bar "Essentially, using nofollow causes us to drop the target links from our overall graph of the web."... Feel free to correct me if anyone knows better, but using nofollow could never harm the target link, it's a "neutral vote" as such... –  Anonymous Jun 20 '11 at 13:07
    
Robots.txt is not appropriate here as these pages can still appear in the index but do not pass on link authority. The "follow, noindex" is the best option here as per Joshak's suggestion. This outlines it seomoz.org/learn-seo/robotstxt –  Sandy Lee Feb 29 '12 at 14:46
add comment

If you're sure Google is reaching the all of your products, and it seems like you are since you're using canonical URLs, then it's a good idea to prevent Google from crawling those pages since all it will do is eat up your bandwidth and server resources.

Using nofollow on any internal links is a good start. I'd also consider using the robots meta tag or x-robots-tag HTTP header to tell them not to index it, too. That way you're sure even if they find those URLs through external linking or otherwise that they will definitely not index those page. Blocking them via robots.txt is also a good thing to since you can use wildcards to catch the pages with query strings, etc.

share|improve this answer
    
Don't use nofollow as it will prevent link authority being passed but the page can still be crawled =) –  Sandy Lee Feb 29 '12 at 14:48
add comment

Since you mentioned the pages can be "exponentially different" you might want to consider using the "follow, noindex" meta tag, the different classifiers can provide Google with additional information about the product so it is potentially beneficial to let them know about the additional content even if you don't want it indexed (this can help for long tail queries).

share|improve this answer
    
This is the best approach =) –  Sandy Lee Feb 29 '12 at 14:48
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.