Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Our robots.txt mistakenly got changed to:

User-agent: *
Disallow: /

This occurred about 2 weeks ago and was changed back yesterday to allow crawling.

Our listed pages in the Googles have decreased substantially, though it looks like Google has already started adding more search results for the site in question since changing the robots.txt back.

Are there long term repercussions of this happening, or will everything get back to normal once the site is crawled again?

share|improve this question
3  
Status Quo will come back. But not in 24hours, give it time –  Martijn Jun 19 at 14:44
add comment

1 Answer 1

up vote 6 down vote accepted

If Google has started adding pages again, then you are fine. We had this happen with a few sites that were pushed from development into production and their robots.txt was not changed.

In fact, unless you are trying to block pages, I recommend removing robots.txt completely. There's no point in saying allow everything in robots.txt when that is the default action.

share|improve this answer
1  
I never really bought the argument that a robots.txt file should exist even if it is empty. That does not seem to make sense. I am sure that the bots we care about are fault tolerant if this was ever a problem. However, there are always bots to block in my view. –  closetnoc Jun 19 at 14:55
2  
Also you can add your sitemap file to the robots.txt, which is not bad –  krokola Jun 19 at 15:02
    
Yea, I've got sitemap in the robots.txt as well. Thanks a bunch for the insight :) –  TomHarrigan Jun 19 at 15:03
1  
Make sure sitemaps and robots.txt are in agreement. I've seen issues where meta info, robots.txt and site maps all told the bots to do different things. –  jeffatrackaid Jun 19 at 16:53
3  
@closetnoc the main reason is to prevent 404 errors in your error log, since search engines will continue to check for the file. –  DisgruntledGoat Jun 19 at 17:10
show 4 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.