Our robots.txt mistakenly got changed to:

User-agent: *
Disallow: /

This occurred about 2 weeks ago and was changed back yesterday to allow crawling.

Our listed pages in the Googles have decreased substantially, though it looks like Google has already started adding more search results for the site in question since changing the robots.txt back.

Are there long term repercussions of this happening, or will everything get back to normal once the site is crawled again?

  • 3
    Status Quo will come back. But not in 24hours, give it time
    – Martijn
    Commented Jun 19, 2014 at 14:44

1 Answer 1


If Google has started adding pages again, then you are fine. We had this happen with a few sites that were pushed from development into production and their robots.txt was not changed.

In fact, unless you are trying to block pages, I recommend removing robots.txt completely. There's no point in saying allow everything in robots.txt when that is the default action.

  • 1
    I never really bought the argument that a robots.txt file should exist even if it is empty. That does not seem to make sense. I am sure that the bots we care about are fault tolerant if this was ever a problem. However, there are always bots to block in my view.
    – closetnoc
    Commented Jun 19, 2014 at 14:55
  • 3
    Also you can add your sitemap file to the robots.txt, which is not bad
    – krokola
    Commented Jun 19, 2014 at 15:02
  • Yea, I've got sitemap in the robots.txt as well. Thanks a bunch for the insight :) Commented Jun 19, 2014 at 15:03
  • 1
    Make sure sitemaps and robots.txt are in agreement. I've seen issues where meta info, robots.txt and site maps all told the bots to do different things. Commented Jun 19, 2014 at 16:53
  • 3
    @closetnoc the main reason is to prevent 404 errors in your error log, since search engines will continue to check for the file. Commented Jun 19, 2014 at 17:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.