4

As I understand, having repetitive content is a detriment to search engine placement.

Given that many websites that use similar or even identical "Terms and Conditions" and "Privacy Policy" pages due to similar legal wording or due to copy & pasting from the same source, would it be a good idea to disallow access to these pages via robots.txt, in order to avoid being penalized for "non-original content"?

Or, on the contrary, could the search engines identify this as circumvention and penalize the site for trying to hide content?

Or does it not matter?

Improve this question
3
  • robots.txt does not prevent Google crawling those pages, it stops them being indexed. Your better off writing your own legal pages for seo purposes
    – Simon Hayter
    Commented Mar 18, 2013 at 23:34
  • Also Google will not punish the entire site for 1-2 duplicate pages... however they won't rank or conbrute to other pages. Forget things like nofoolllow, noindex, robots - best method is writing your own
    – Simon Hayter
    Commented Mar 18, 2013 at 23:42
  • 1
    Bybe's wrong about robots.txt. It does prevent crawling, but doesn't prevent the blocked content being indexed (if the content in question is linked to).
    – GDVS
    Commented Mar 20, 2013 at 23:03

2 Answers 2

Reset to default
1

On the one hand, Google probably doesn't want to index these pages on your site. Most of the words in these pages aren't going to be relevant to the rest of your site. If somebody searches for these words, your site will not make a great landing page.

On the other hand, so many sites have these types of pages, that it isn't going to cause a problem for Google to let them crawl them. Other sites allow them to be crawled, your site can too. Here are some good reasons that you might want to have them crawled:

  • There are some terms for which you would like have them show up in search: "contact <brand>" or "<brand> TOS".
  • They often make good "site links" (the links that Google shows under your listing when somebody searches for your brand).
  • Just having these pages is the mark of a reputable site. You certainly want Google to know that you have these pages so that they consider your site reputable.

Your site wouldn't be penalized for putting them in robots.txt, but overall having these pages crawlable will help your site.

Improve this answer
1

No, it won't matter that much.

Matt Cutts recommends not focusing too much on this, but on your content: http://www.youtube.com/watch?v=unEML5n5vTo

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.