5

Our robots.txt mistakenly got changed to: 

User-agent: *
Disallow: /

This occurred about 2 weeks ago and was changed back yesterday to allow crawling.

Our listed pages in the Googles have decreased substantially, though it looks like Google has already started adding more search results for the site in question since changing the robots.txt back.

Are there long term repercussions of this happening, or will everything get back to normal once the site is crawled again?

Improve this question
  • 3
    Status Quo will come back. But not in 24hours, give it time – Martijn Jun 19 '14 at 14:44
6

If Google has started adding pages again, then you are fine. We had this happen with a few sites that were pushed from development into production and their robots.txt was not changed.

In fact, unless you are trying to block pages, I recommend removing robots.txt completely. There's no point in saying allow everything in robots.txt when that is the default action.

Improve this answer
  • 1
    I never really bought the argument that a robots.txt file should exist even if it is empty. That does not seem to make sense. I am sure that the bots we care about are fault tolerant if this was ever a problem. However, there are always bots to block in my view. – closetnoc Jun 19 '14 at 14:55
  • 3
    Also you can add your sitemap file to the robots.txt, which is not bad – krokola Jun 19 '14 at 15:02
  • Yea, I've got sitemap in the robots.txt as well. Thanks a bunch for the insight :) – TomHarrigan Jun 19 '14 at 15:03
  • 1
    Make sure sitemaps and robots.txt are in agreement. I've seen issues where meta info, robots.txt and site maps all told the bots to do different things. – jeffatrackaid Jun 19 '14 at 16:53
  • 3
    @closetnoc the main reason is to prevent 404 errors in your error log, since search engines will continue to check for the file. – DisgruntledGoat Jun 19 '14 at 17:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.