I developed a page under the following URL: example.org/dev

Since the development is finished, I moved the page to the root at example.org/

For some reason, Google indexed the page during development. Hence, Google shows results leading to example.org/dev, even if this yields a (Wordpress) 404.

How can I tell Google to remove these pages? What I did for now, is to create a robots.txt with the following content:

# robots.txt
User-agent: * 
Disallow: /dev/

Is there any other/better thing I can do?

  • nope. just a copy/paste error. update now.
    – beta
    Feb 15 '16 at 12:31
  • 1
    Your robots.txt should be enough. Google checks the robot.txt file after 24 hours unless they have not hit your site. In other words, if your site is not normally hit every day, Google will notice the change to the robot.txt file the next time they visit. Then they will apply it fairly quickly. It still can take a while, however, the robots.txt seems to be the fasted method.
    – closetnoc
    Feb 15 '16 at 16:46

The best thing to do is redirect your entire dev directory to your main domain. The next time that Googlebot crawls it, it will find it moved. It will then remove the dev pages from the index and only include the main site.

You can implement the redirect in .htaccess like:

Redirect permanent /dev/ http://example.org/
  • 1
    When you implement the redirect you'll also need to unblock it in robots.txt.
    – MrWhite
    Feb 15 '16 at 23:01
  • okay, so which option is better? a) modify .htaccess (and unblock /dev/ in robots.txt) or b) no modification of .htaccess, but disallow in robots.txt?
    – beta
    Feb 16 '16 at 8:12
  • If Google can't crawl it, it will probably remove it from the search index eventually, but it can be slow to do so. Redirects should be faster (a couple weeks). Feb 16 '16 at 9:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.