1

I have a few domains that I used for developing and testing websites. As it usually has the actual content of the production site, I don't want Google to index those domains ever.

I don't want to use robots.txt as a solution. I'm constantly deleting and reinstalling everything on these domains, robots.txt included. I don't want to set some sort of HTTP status code, because 200 OK is what I need during testing. Password protecting the site also is a pain because clients get confused, it prevents me from certain testing, etc.

Is there something in Webmaster Tools that I can use to tell Google not to index the site? Also, how do I remove any and all of the pages Google has indexed so far?

1

Honestly if you recreate everything each time, I do not see why the robots.txt is not just reconstructed too. The part to prevent Googlebot is static and just needs to be there when you restore the site.

While Google Search console will not help getting the site in index, it will confirm if you have done so by listing the pages as Blocked Resources.

There several ways to block resources:

  • Use tags like: <meta name="robots" content="noindex"> See info by Google.
  • robots.txt which you want to avoid but it designed exactly for that and, unlike using Meta goes not have Googlebot fetch to discard, saving you some bandwidth and requests.
  • Sent a 403 Forbidden just to Googlebot (or bots in general). You are generally not testing as a bot or at least not Googlebot, so that status will be hidden from your users and you are free to sent a 200 OK status to humans.
  • robots.txt seems the way to go. I'm curious about how to send a 403 Forbidden to Googlebot. How would I do that? – WebMasterNewbie Dec 21 '17 at 10:50
  • There a few ways, so the one you choose depends on your technology stack. With Apache you can add a rule in the .htaccess that returns, just like in this question, except the user-agent is a googlebot. Using PHP, you can send a response based on the user-agent but it must be in your boiler-plate code that gets added to the start of all your pages. – Itai Dec 21 '17 at 15:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.