We have a non-production version of our site that we want to prevent from being crawled. At some point in the past a non-production site was crawled and we still have search results pointing to it. I have a CNAME setup that points that particular DNS entry to our production site so that we do not lose customers or have issues with them going to one of our non-production sites.
The previous sysadmin was using firewall rules to block traffic, but apparently this wasn't sufficient enough, since a Google crawler indexed this page. I have moved this particular non-prod instance to a new DNS name and have added a
robots.txt to block all crawls/bots:
User-agent: * Disallow: /
Is the above
robots.txt sufficient to prevent the website from being crawled by search indexers? I also plan to update/change the firewall rule and put the non-production sites behind a VPN, however that requires work beyond a simple rule, as our VPN is lacking. Until we get to that point, I'd like to know if my
robots.txt will suffice or if I should take more drastic measures to hide the site.