2

We have a non-production version of our site that we want to prevent from being crawled. At some point in the past a non-production site was crawled and we still have search results pointing to it. I have a CNAME setup that points that particular DNS entry to our production site so that we do not lose customers or have issues with them going to one of our non-production sites.

The previous sysadmin was using firewall rules to block traffic, but apparently this wasn't sufficient enough, since a Google crawler indexed this page. I have moved this particular non-prod instance to a new DNS name and have added a robots.txt to block all crawls/bots:

User-agent: *
Disallow: /

Is the above robots.txt sufficient to prevent the website from being crawled by search indexers? I also plan to update/change the firewall rule and put the non-production sites behind a VPN, however that requires work beyond a simple rule, as our VPN is lacking. Until we get to that point, I'd like to know if my robots.txt will suffice or if I should take more drastic measures to hide the site.

Improve this question
3
  • Do you want to prevent the site from getting indexed in search engines or are you fine with it getting indexed even if it isn't crawled?
    – Stephen Ostermiller ♦
    Nov 23, 2022 at 10:07
  • @StephenOstermiller I'm trying to avoid this site every showing up on a search engine. We don't care about it being public, but we don't want Google/Bing/etc ever showing results for this site.
    – Ian Pringle
    Nov 23, 2022 at 11:37
  • 2
    Then you are not asking the right question. Preventing the site from getting crawled will not always prevent it from showing up in search engines.
    – Stephen Ostermiller ♦
    Nov 23, 2022 at 12:45

1 Answer 1

Reset to default
2

No it is not enough at all.

By the means of robots.txt you can only block robot requests that are comming directly to your site.
But the door is still open to crawlers that are landing to your site with follow links.

Best practice for not getting indexed are:
1-To make your subdomain or directory password protected (If applicable)
2-Using X-Robots-Tag with noindex response for the desigred directory.
3-Using below code on every page you dont want to be indexed.

<meta name="robots" content="noindex">


Ref: https://developers.google.com/search/docs/crawling-indexing/block-indexing

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.