I'm having a lot of trouble with something that should be simple. I don't want my development site indexed by Google. According to Google's instructions, and just about everything else I can find says to block this through robots.txt. I set this up correctly months ago, but my development sites still show up. When I log into Google Webmasters and use their robots.txt tester, it shows my site and every URL I've tested should be completely blocked.

User-agent: * Disallow: /

As an extra measure, I added a simple .htpasswd to the website too, but cached pages still showed up. I went through each and every page and removed it using the Google removal tool - most of them disappeared temporarily but week after week more and more pages pop up as being indexed. I've tried to use Wordpress settings to disallow robots, and I've tried including page-specific code in the header too. This has been going on for quite some time.

Long story short, this is starting to drive me crazy as I feel like I've tried everything I can to stop Google from indexing my site, but they won't. Can anyone think of what I could have done wrong?

  • 1
    robots.txt stops crawling, not indexing. Either use x-botos-tag: noindex or block via basic authentication. – John Conde Oct 21 '15 at 15:32
  • 1
    As a side note, if you use the noindex method, you will have to allow Google to crawl your site again to see the noindex. It is a bit of a catch 22, but it works as John says just fine. – closetnoc Oct 21 '15 at 15:39
  • I've already added these: <meta name="robots" content="noindex"> <meta name="googlebot" content="noindex"> – bradfordg Oct 21 '15 at 16:11
  • What is the description in the SERPs of these indexed pages? – DocRoot Oct 21 '15 at 17:07