I have a staging server that accidentally got indexed by google. I want to remove ALL pages from google for that site: i.e. stagingserver.mycompany.com. I added a robots.txt to the root of the site with following content:
User-agent: * Disallow: /
After several weeks, none of the pages have been removed. When I google:
I get pages and pages of results. I went into Webmaster tools. I created many removal requests, and they only work for those specific pages. When I add a directory removal:
it takes a few hours for the status to turn from pending to removed, but all the unwanted pages are still not removed. It seems that there's no way to remove ALL pages from google. Is this right?
I can't send a 404 for every page because this is a development site, that is being worked on and reviewed by clients.