I need to block pages such as privacy policy, terms and conditions, and guarantees from being crawled and indexed by Google. They might be unoriginal. I use them for my multiple websites and they have no content relevant to be ranked by search engines.

Their links are like https://example.com/privacy.php

I don't want to mess up and end up being penalized by Google

  • 2
    What is this "different way" you refer to in the question title? What you describe in the question would seem to be exactly what robots.txt is used for, in a normal way. So, what is the problem exactly?
    – MrWhite
    Mar 14 '17 at 14:41
  • @OskarSkog i don't understand what i should do..please give me a step by step on how to do it Mar 16 '17 at 13:51
  • @w3dk I just need a step by step way of creating the txt file ...if you can send me the sample id appreciate...i don't know the best way to do it,either robots.txt or noindex- i simply don't know Mar 16 '17 at 18:34
  • Google understands that pages like these are often similar or the across many sites. It won't cause any issues to allow them to be crawled. Google generally only ranks them for queries like "mysite privacy policy". Google may use the existence of contact forms and privacy policies as a ranking signal, so it is a good idea to let Google crawl them. Apr 19 '17 at 12:11

Just go to your C panel>>public_html>>site directory and create a file named "robots.txt" in it paste the following code

User-Agent: *
Disallow: /privacy.php/
Disallow: /terms.php/
Sitemap: http://www.xxxxxxx.xxx/sitemap.xml 

Change the links above to match up with your site. With the above, google bots can never crawl any information in the "Disallow" field.

  • 1. I think the trailing slashes after the PHP files is erroneous. 2. Google can crawl that information if it chooses to, they just choose not to do so. So saying "never" isn't quite accurate.
    – John Conde
    Mar 17 '17 at 0:17
  • @Yesokay that's great. I appreciate. Do you agree with "I think the trailing slashes after the PHP files is erroneous" Mar 20 '17 at 9:00
  • @Nina , just remove the slash, the rest will work out fine
    – Yesokay
    Mar 22 '17 at 18:05
  • @Yesokay "The home page for this site is disallowed by robots.txt, please allow it" How can i do this yet I have not disallowed my homepage, what might be wrong ? May 24 '17 at 15:06
  • @Nina, can you copy and paste what's in your robots.txt file and a link to the site?
    – Yesokay
    Jul 8 '17 at 20:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.