Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

As I understand, having repetitive content is a detriment to search engine placement.

Given that many websites that use similar or even identical "Terms and Conditions" and "Privacy Policy" pages due to similar legal wording or due to copy & pasting from the same source, would it be a good idea to disallow access to these pages via robots.txt, in order to avoid being penalized for "non-original content"?

Or, on the contrary, could the search engines identify this as circumvention and penalize the site for trying to hide content?

Or does it not matter?

share|improve this question
    
robots.txt does not prevent Google crawling those pages, it stops them being indexed. Your better off writing your own legal pages for seo purposes –  bybe Mar 18 '13 at 23:34
    
Also Google will not punish the entire site for 1-2 duplicate pages... however they won't rank or conbrute to other pages. Forget things like nofoolllow, noindex, robots - best method is writing your own –  bybe Mar 18 '13 at 23:42
1  
Bybe's wrong about robots.txt. It does prevent crawling, but doesn't prevent the blocked content being indexed (if the content in question is linked to). –  GDav Mar 20 '13 at 23:03
add comment

2 Answers

On the one hand, Google probably doesn't want to index these pages on your site. Most of the words in these pages aren't going to be relevant to the rest of your site. If somebody searches for these words, your site will not make a great landing page.

On the other hand, so many sites have these types of pages, that it isn't going to cause a problem for Google to let them crawl them. Other sites allow them to be crawled, your site can too. Here are some good reasons that you might want to have them crawled:

  • There are some terms for which you would like have them show up in search: "contact <brand>" or "<brand> TOS".
  • They often make good "site links" (the links that Google shows under your listing when somebody searches for your brand).
  • Just having these pages is the mark of a reputable site. You certainly want Google to know that you have these pages so that they consider your site reputable.

Your site wouldn't be penalized for putting them in robots.txt, but overall having these pages crawlable will help your site.

share|improve this answer
add comment

No, it won't matter that much.

Matt Cutts recommends not focusing too much on this, but on your content: http://www.youtube.com/watch?v=unEML5n5vTo

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.