3

Nine days ago, I got a message Google Webmaster Tools:

Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt.

Well, but I don't have a robots.txt on that site, because robots.txt is optional and I want the whole site to be crawled. So why do I get this error message?

Perhaps of interest: The Google Webmaster tools home page lists www.realitybuilder.com and realitybuilder.com. I don't know how that happened, but realitybuilder.com redirects to www.realitybuilder.com, so it should not be necessary to have it listed. I now deleted the entry for realitybuilder.com. Could that have caused the problem?

Missing robots.txt error message

Improve this question
10
  • 1
    Have you ever had a robots.txt file on your site?
    – MrWhite
    Feb 8, 2013 at 14:46
  • @w3d: I don't think so. For that site it wouldn't make sense anyhow. It's a one page web app.
    – feklee
    Feb 8, 2013 at 15:03
  • One suspicion by me is that the entire site was indeed offline for some time, and Googlebot got a 500. Google could just provide more details, and I wouldn't need to ask here.
    – feklee
    Feb 8, 2013 at 15:06
  • 1
    It is quite usual to have both www.example.com and example.com listed in GWT. You must have added both for both to be present. You say that www. is the main site (that is redirected to), however, your error appears to relate to the non-www version?
    – MrWhite
    Feb 8, 2013 at 16:34
  • @w3d Yes, the error applies to the non-www version, as you can see in the screenshot.
    – feklee
    Feb 8, 2013 at 17:09

3 Answers 3

Reset to default
2

I'm not sure why webmaster tools does this but I've had a similar problem with my site. When it was in development, I blocked it using the robots.txt file, then removed the block when it went live, but webmaster tools took a while to update itself.

What I'd recommend is to do a fetch as Googlebot and submit all pages, that should get Google looking at your site again quicker.

One last thing, your right robots.txt is optional but it might help the search engines understand better in you make a robots.txt file and set it to:

User-Agent: *
Disallow:

Which is just like saying all pages are allowed.

Improve this answer
4
  • Only, I highly doubt that I ever put a robots.txt on that site. And, in any case, I didn't update the site in many months.
    – feklee
    Feb 8, 2013 at 15:08
  • I now did the fetch as Google. Please note that while your answer is helpful, I cannot mark it as accepted since it doesn't answer the why.
    – feklee
    Feb 8, 2013 at 15:23
  • 1
    as to the why im not sure, reading over this question though i rember i asked somthing extremly similar a while back, have a look at this it may be helpful - webmasters.stackexchange.com/questions/38099/…
    – sam
    Feb 8, 2013 at 17:17
  • Thanks, @sam, for pointing me to your question. There is a big difference, though: You get an error message because of an existing robots.txt, I get one because of a missing robots.txt.
    – feklee
    Feb 8, 2013 at 17:38
2

This error occur when your robots.txt file exists but is unreachable. Your site should return 200 HTTP status if the file exists or 404 if it doesn't, otherwise you would face such message from google.

Before Googlebot crawls your site, it accesses your robots.txt file to determine if your site is blocking Google from crawling any pages or URLs. If your robots.txt file exists but is unreachable (in other words, if it doesn’t return a 200 or 404 HTTP status code), we’ll postpone our crawl rather than risk crawling URLs that you do not want crawled. When this happens, Googlebot will return to your site and crawl it as soon as we can successfully access your robots.txt file.

Improve this answer
2
  • 1
    It would be best to include a link to the source of that quote.
    – Stephen Ostermiller ♦
    Dec 12, 2014 at 18:11
  • 1
    @StephenOstermiller Done, thank you for your notice.
    – Moein Nourmohammadi
    Dec 13, 2014 at 2:51
0

If you don’t have a robot.txt file and Googlebot encountered 1 error, then I think you have to define your robots.txt.

Robots.txt is a simple text file. If you want your entire website to be crawled, then you can define your robots.txt file as below.

User-agent: *
Disallow:

In this file User-agent: * means this section applies to all robots and the Disallow: tells the robot that it should visit all the pages on the site.

Upload your robots.txt file with this code and after few days see that Google webmaster tool shows any error or not.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.