I was making a site for a friend of mine and he wanted to be able to see my progress as I worked on the site, so I decided to put the site on a server on my computer and enable access by a domain name registered to me.

It turns out that I forgot to set up a robots.txt file for the site and somehow Google indexed the site.

My question is: What do I do now?

As I understand it, Google doesn't like duplicate content and my friend could have problems when I upload the new site to his server.

Right now his current site, which only has a work in progress page, is first on Google when searching for relevant keywords and I really really don't want to damage that.

Is there anything else I need to be concerned about?

  • 2
    +1 for getting the site on top for relevant keywords without any efforts! :) lol i wish that would happen with me sometime. :( Mar 8, 2011 at 6:53

4 Answers 4


1) Immediately block the site you have up now using robots.txt

2) On the development site use canonical URLs to point to the new site so that site gets credit for the content. They do work cross-domain. This meta tag may also help.

3) after you launch use 301 redirects from the development site to the new site so Google knows those pages have moved to new URLs

  • Good to know about the canonical tag working across domains as I thought it was just within the same domain - Is that a change from the past? Mar 9, 2011 at 2:22

Add the robots.txt file to your server then request the pages to be removed from Google's index using Google Webmaster Tools. You should then set up 301 redirects to the new site when it goes live.

  • Do you have any idea how long it takes for removal request to be processes using Google Webmaster Tools? I placed request several hours ago and it is still waiting. Is that normal?
    – AndrejaKo
    Mar 8, 2011 at 14:17
  • 1
    When I used it before on a hacked site it took a couple of days for all the links to be removed. It will be more than a few hours anyway. Mar 9, 2011 at 2:23

In the future, you should set up Basic Authentication in Apache or IIS for test pages. This will not only prevent indexing (even from spiders that don't respect robots.txt) but will also prevent random users from getting to your site and seeing what you don't want to be public yet.

  • 1
    Probably something we all should do more often with development sites... Mar 9, 2011 at 2:28

Remove a page or site from Google's search results - http://www.google.com/support/webmasters/bin/answer.py?answer=164734

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.