Nine days ago, I got a message Google Webmaster Tools:

Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt.

Well, but I don't have a robots.txt on that site, because robots.txt is optional and I want the whole site to be crawled. So why do I get this error message?

Perhaps of interest: The Google Webmaster tools home page lists www.realitybuilder.com and realitybuilder.com. I don't know how that happened, but realitybuilder.com redirects to www.realitybuilder.com, so it should not be necessary to have it listed. I now deleted the entry for realitybuilder.com. Could that have caused the problem?

Missing robots.txt error message

  • 1
    Have you ever had a robots.txt file on your site? – MrWhite Feb 8 '13 at 14:46
  • @w3d: I don't think so. For that site it wouldn't make sense anyhow. It's a one page web app. – feklee Feb 8 '13 at 15:03
  • One suspicion by me is that the entire site was indeed offline for some time, and Googlebot got a 500. Google could just provide more details, and I wouldn't need to ask here. – feklee Feb 8 '13 at 15:06
  • 1
    It is quite usual to have both www.example.com and example.com listed in GWT. You must have added both for both to be present. You say that www. is the main site (that is redirected to), however, your error appears to relate to the non-www version? – MrWhite Feb 8 '13 at 16:34
  • @w3d Yes, the error applies to the non-www version, as you can see in the screenshot. – feklee Feb 8 '13 at 17:09

I'm not sure why webmaster tools does this but I've had a similar problem with my site. When it was in development, I blocked it using the robots.txt file, then removed the block when it went live, but webmaster tools took a while to update itself.

What I'd recommend is to do a fetch as Googlebot and submit all pages, that should get Google looking at your site again quicker.

One last thing, your right robots.txt is optional but it might help the search engines understand better in you make a robots.txt file and set it to:

User-Agent: *

Which is just like saying all pages are allowed.

  • Only, I highly doubt that I ever put a robots.txt on that site. And, in any case, I didn't update the site in many months. – feklee Feb 8 '13 at 15:08
  • I now did the fetch as Google. Please note that while your answer is helpful, I cannot mark it as accepted since it doesn't answer the why. – feklee Feb 8 '13 at 15:23
  • 1
    as to the why im not sure, reading over this question though i rember i asked somthing extremly similar a while back, have a look at this it may be helpful - webmasters.stackexchange.com/questions/38099/… – sam Feb 8 '13 at 17:17
  • Thanks, @sam, for pointing me to your question. There is a big difference, though: You get an error message because of an existing robots.txt, I get one because of a missing robots.txt. – feklee Feb 8 '13 at 17:38

This error occur when your robots.txt file exists but is unreachable. Your site should return 200 HTTP status if the file exists or 404 if it doesn't, otherwise you would face such message from google.

Before Googlebot crawls your site, it accesses your robots.txt file to determine if your site is blocking Google from crawling any pages or URLs. If your robots.txt file exists but is unreachable (in other words, if it doesn’t return a 200 or 404 HTTP status code), we’ll postpone our crawl rather than risk crawling URLs that you do not want crawled. When this happens, Googlebot will return to your site and crawl it as soon as we can successfully access your robots.txt file.

  • 1
    It would be best to include a link to the source of that quote. – Stephen Ostermiller Dec 12 '14 at 18:11
  • 1
    @StephenOstermiller Done, thank you for your notice. – Moein Nourmohammadi Dec 13 '14 at 2:51

If you don’t have a robot.txt file and Googlebot encountered 1 error, then I think you have to define your robots.txt.

Robots.txt is a simple text file. If you want your entire website to be crawled, then you can define your robots.txt file as below.

User-agent: *

In this file User-agent: * means this section applies to all robots and the Disallow: tells the robot that it should visit all the pages on the site.

Upload your robots.txt file with this code and after few days see that Google webmaster tool shows any error or not.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.