Dev site indexed by Google

Question

I was developing a new version of our site on a subdomain (dev.oursite.com) and the robots.txt with the Disallow got replaced at some point, so the site got indexed by Google and other search engines. It's not showing up high in results or anything, but it has all duplicate content and I'd rather it not be there.

The subdomain is now gone and I have a 301 that redirects each page from dev.oursite.com/page-name to http://oursite.com/page-name.

Is there anything else I need to do to get the dev site removed from showing up in Google? Will it disappear by itself eventually?

Answer 1

Check out the URL removal tool in Google Webmaster Tools. I'd also 404 the pages instead of redirecting them to get them removed faster, in the future beyond robots.txt you could drop in the rel="canonical" to make sure Google knows the dev site is just a copy of the main site and is not to be indexed.

Answer 2

I'm always concerned about development sites becoming indexed. I don't trust robots.txt or meta noindex, while I do use them I also password protect the sites if it's not to inconvenient. But another option is to use .htaccess and deny access to everyone except IP's within your company and for co-workers and developers. Just add their class C.

Answer 3

In addition to the correct answer provided by Joshak I would like to provide you with a tip on how this can be prevented.

What I have done to solve this exact same issue is to force the robot.txt in the Apache httpd definition of the vhost. This way there is no way the "disallow" can vanish or be changed by any code in the website under development. My vhost definitions all look something like this:

<VirtualHost *:80>
    DocumentRoot /var/www/html/Hosting/test.example.com/newsite.com
    ServerName newsite.com.test.example.com
    ServerAlias *.newsite.com.test.example.com
    UseCanonicalName on
    RewriteEngine on
    RewriteRule ^/robots.txt /var/www/no-indexing-robots.txt [NC,L]
</VirtualHost>