3

I'm not sure whether I should add my website's privacy policy to my robots.txt file or not. I want to follow good SEO practice. I have no reason to hide my privacy policy.

Arguments against: I want to be visible and not seen to be hiding legal pages. It's unlikely, but some users might genuinely want to find it in a search engine.

Arguments for: solicitors/lawyers tend to reuse these documents so the text I've been supplied may not be unique. The page isn't really one that's important for people to land on from a search engine.

Any advice?

5

Your arguments against not blocking the indexing of these legal pages are valid. Matt Cuts did a video about this very topic here in 2009:

https://www.youtube.com/watch?v=unEML5n5vTo

While it's not a clear "yes" or "no" from Google, the second important question to ask is "are these pages important to my users?". If yes, Google needs to find them.

Based on our research we've written about this topic on our blog, we find that these are common legal pages. Google knows that these policies are required by law, that the legal pages are not content-rich and that you're most likely to have similar content as other websites on these pages.

If you do decide to block these pages from being indexed, use meta robots tag instead of the robots.txt:

<meta name="robots" content="noindex, nofollow" />

Any error in the robots.txt might get your entire website unindexed by Google. The meta robots tag is the safe option to manage this.

  • I'd go so far as to say that Google expects to find these pages on high quality sites. You should allow these legal documents to be crawled and indexed because Google already knows how to deal with them and they are a quality signal. – Stephen Ostermiller Dec 19 '17 at 10:46
  • Small point: an error in robots.txt may stop your entire website from being crawled, but won't in itself get the site de-indexed. Important difference. – GDav Dec 20 '17 at 7:49
1

Avoid making any reference to the privacy or terms and conditions page in the robots.txt. The Best SEO practice would be to create a meta robots with noindex, nofollow instructions in the head section as follow:

<meta name="robots" content="noindex,nofollow">

This way your privacy page won’t be indexed.

You need to make this page accessible by the user by creating an internal link in the footer. This way search engines will be able to find the page too and then find the noindex, nofollow instructions.

You can use other approaches, but to me is the best one. Make your website always crawlable and use robots file to safe yourself and others crawling resources.

  • If it's "noindex,nofollow", why not putting it in the robots.txt? It would be good to understand the reason behind that. – keith Dec 19 '17 at 9:14
  • @keith robots file won’t guarantee avoiding pages to get indexed. If some other source link to a particular page, that page could still get indexed. To me it is preferable to use it to set rules to avoid bots crawling folders, particular URL patterns, authorised only website sections, etc. Sometimes bots might or might not respect its instructions, others claim that it is archaic. As I said, use it to save bots some work and for larger sites to save some bandwidth or server resources. – Raul Reyes Dec 19 '17 at 12:33
  • Thanks that makes a lot of sense (I've upvoted as this was very useful). – keith Dec 19 '17 at 14:12
  • @keith no worries I am glad that I could clarify why to use the meta tag :) – Raul Reyes Dec 19 '17 at 19:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.