I need to block pages such as privacy policy, terms and conditions, and guarantees from being crawled and indexed by Google. They might be unoriginal. I use them for my multiple websites and they have no content relevant to be ranked by search engines.

Their links are like https://example.com/privacy.php

I don't want to mess up and end up being penalized by Google

  • 2
    What is this "different way" you refer to in the question title? What you describe in the question would seem to be exactly what robots.txt is used for, in a normal way. So, what is the problem exactly?
    – MrWhite
    Commented Mar 14, 2017 at 14:41
  • @OskarSkog i don't understand what i should do..please give me a step by step on how to do it Commented Mar 16, 2017 at 13:51
  • @w3dk I just need a step by step way of creating the txt file ...if you can send me the sample id appreciate...i don't know the best way to do it,either robots.txt or noindex- i simply don't know Commented Mar 16, 2017 at 18:34
  • Google understands that pages like these are often similar or the across many sites. It won't cause any issues to allow them to be crawled. Google generally only ranks them for queries like "mysite privacy policy". Google may use the existence of contact forms and privacy policies as a ranking signal, so it is a good idea to let Google crawl them. Commented Apr 19, 2017 at 12:11

1 Answer 1


Just go to your C panel>>public_html>>site directory and create a file named "robots.txt" in it paste the following code

User-Agent: *
Disallow: /privacy.php/
Disallow: /terms.php/
Sitemap: http://www.xxxxxxx.xxx/sitemap.xml 

Change the links above to match up with your site. With the above, google bots can never crawl any information in the "Disallow" field.

  • 1. I think the trailing slashes after the PHP files is erroneous. 2. Google can crawl that information if it chooses to, they just choose not to do so. So saying "never" isn't quite accurate.
    – John Conde
    Commented Mar 17, 2017 at 0:17
  • @Yesokay that's great. I appreciate. Do you agree with "I think the trailing slashes after the PHP files is erroneous" Commented Mar 20, 2017 at 9:00
  • @Nina , just remove the slash, the rest will work out fine
    – Yesokay
    Commented Mar 22, 2017 at 18:05
  • @Yesokay "The home page for this site is disallowed by robots.txt, please allow it" How can i do this yet I have not disallowed my homepage, what might be wrong ? Commented May 24, 2017 at 15:06
  • @Nina, can you copy and paste what's in your robots.txt file and a link to the site?
    – Yesokay
    Commented Jul 8, 2017 at 20:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.