I was developing a new version of our site on a subdomain (dev.oursite.com) and the robots.txt with the Disallow got replaced at some point, so the site got indexed by Google and other search engines. It's not showing up high in results or anything, but it has all duplicate content and I'd rather it not be there.

The subdomain is now gone and I have a 301 that redirects each page from dev.oursite.com/page-name to http://oursite.com/page-name.

Is there anything else I need to do to get the dev site removed from showing up in Google? Will it disappear by itself eventually?


Check out the URL removal tool in Google Webmaster Tools. I'd also 404 the pages instead of redirecting them to get them removed faster, in the future beyond robots.txt you could drop in the rel="canonical" to make sure Google knows the dev site is just a copy of the main site and is not to be indexed.


I'm always concerned about development sites becoming indexed. I don't trust robots.txt or meta noindex, while I do use them I also password protect the sites if it's not to inconvenient. But another option is to use .htaccess and deny access to everyone except IP's within your company and for co-workers and developers. Just add their class C.

  • 1
    This is the best way. 403 everyone but a series of addresses or address blocks. The rest of the world goes away and only those who need to see it can. After the fact for this question, but good to know for the future. You may need a privately accessible dev site even after launch for upgrade trials, custom programming from third party developers, etc. – Fiasco Labs Sep 15 '13 at 17:28

In addition to the correct answer provided by Joshak I would like to provide you with a tip on how this can be prevented.

What I have done to solve this exact same issue is to force the robot.txt in the Apache httpd definition of the vhost. This way there is no way the "disallow" can vanish or be changed by any code in the website under development. My vhost definitions all look something like this:

<VirtualHost *:80>
    DocumentRoot /var/www/html/Hosting/test.example.com/newsite.com
    ServerName newsite.com.test.example.com
    ServerAlias *.newsite.com.test.example.com
    UseCanonicalName on
    RewriteEngine on
    RewriteRule ^/robots.txt /var/www/no-indexing-robots.txt [NC,L]

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.