0

I have a staging server that accidentally got indexed by google. I want to remove ALL pages from google for that site: i.e. stagingserver.mycompany.com. I added a robots.txt to the root of the site with following content:

User-agent: *
Disallow: /

After several weeks, none of the pages have been removed. When I google:

site:stagingserver.mycompany.com

I get pages and pages of results. I went into Webmaster tools. I created many removal requests, and they only work for those specific pages. When I add a directory removal:

http://stagingserver.mycompany.com/*

it takes a few hours for the status to turn from pending to removed, but all the unwanted pages are still not removed. It seems that there's no way to remove ALL pages from google. Is this right?

NOTE:

I can't send a 404 for every page because this is a development site, that is being worked on and reviewed by clients.

  • 2
    Basically send a 404 for all of the pages you want removed – John Conde Jun 23 '15 at 18:41
  • 404 is not an option as this is a working development site. Do you mean create a 404 just for the googlebot? – David Tay Jun 23 '15 at 18:49
  • 1
    You should still be able to send a 404 "HTTP status code" - you don't need to send a complete error document, just send the normal page. However, this is probably irrelevant if you are already blocking with robots.txt. Is the * suffix required on the directory removal request? Where have you seen this documented? – MrWhite Jun 23 '15 at 19:37
  • In your case, blocking the site using robots.txt will prevent the 404 from taking effect since the page will not be fetched. Using a 404 is not actually faster than using the robots.txt file. Using the robots.txt is the fastest way to have the pages removed from the index. No matter what you do, you will still have to be patient. Search engines are very far from being real-time. – closetnoc Jun 23 '15 at 21:07