Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Nine days ago, I got a message Google Webmaster Tools:

Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt.

Well, but I don't have a robots.txt on that site, because robots.txt is optional and I want the whole site to be crawled. So why do I get this error message?

Perhaps of interest: The Google Webmaster tools home page lists www.realitybuilder.com and realitybuilder.com. I don't know how that happened, but realitybuilder.com redirects to www.realitybuilder.com, so it should not be necessary to have it listed. I now deleted the entry for realitybuilder.com. Could that have caused the problem?

Missing robots.txt error message

share|improve this question
1  
Have you ever had a robots.txt file on your site? –  w3d Feb 8 '13 at 14:46
    
@w3d: I don't think so. For that site it wouldn't make sense anyhow. It's a one page web app. –  feklee Feb 8 '13 at 15:03
    
One suspicion by me is that the entire site was indeed offline for some time, and Googlebot got a 500. Google could just provide more details, and I wouldn't need to ask here. –  feklee Feb 8 '13 at 15:06
1  
It is quite usual to have both www.example.com and example.com listed in GWT. You must have added both for both to be present. You say that www. is the main site (that is redirected to), however, your error appears to relate to the non-www version? –  w3d Feb 8 '13 at 16:34
    
@w3d Yes, the error applies to the non-www version, as you can see in the screenshot. –  feklee Feb 8 '13 at 17:09
show 5 more comments

2 Answers

I'm not sure why webmaster tools does this but I've had a similar problem with my site. When it was in development, I blocked it using the robots.txt file, then removed the block when it went live, but webmaster tools took a while to update itself.

What I'd recommend is to do a fetch as Googlebot and submit all pages, that should get Google looking at your site again quicker.

One last thing, your right robots.txt is optional but it might help the search engines understand better in you make a robots.txt file and set it to:

User-Agent: *
Disallow: 

Which is just like saying all pages are allowed.

share|improve this answer
    
Only, I highly doubt that I ever put a robots.txt on that site. And, in any case, I didn't update the site in many months. –  feklee Feb 8 '13 at 15:08
    
I now did the fetch as Google. Please note that while your answer is helpful, I cannot mark it as accepted since it doesn't answer the why. –  feklee Feb 8 '13 at 15:23
1  
as to the why im not sure, reading over this question though i rember i asked somthing extremly similar a while back, have a look at this it may be helpful - webmasters.stackexchange.com/questions/38099/… –  sam Feb 8 '13 at 17:17
    
Thanks, @sam, for pointing me to your question. There is a big difference, though: You get an error message because of an existing robots.txt, I get one because of a missing robots.txt. –  feklee Feb 8 '13 at 17:38
add comment

If you don’t have a robot.txt file and Googlebot encountered 1 error, then I think you have to define your robots.txt.

Robots.txt is a simple text file. If you want your entire website to be crawled, then you can define your robots.txt file as below.

User-agent: *
Disallow:

In this file User-agent: * means this section applies to all robots and the Disallow: tells the robot that it should visit all the pages on the site.

Upload your robots.txt file with this code and after few days see that Google webmaster tool shows any error or not.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.