Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

In my blog’s Webmaster Tools, there is a notification in Crawl Errors section, that is Google couldn't crawl your site because we were unable to access the robots.txt file.

My blog’s robots.txt file is:

User-agent: Mediapartners-Google
Disallow: 
User-agent: *
Disallow: /search
Allow: /

Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED

I don’t think the above file details are wrong but I don’t understand why I received such dangerous notification? How to fix this issue?

See my robots.txt file.

share|improve this question
    
1. "User-agent: Mediapartners-Google Disallow:" What does this section mean? 2. "Google couldn't crawl your site because we were unable to access the robots.txt file." Probably, it means your server did not return your robots.txt file while Google had tried to get it. –  Marian Jul 19 '13 at 6:19
    
It only disallow the media-partners of Google like adsence.. Does it is wrong? –  Sathiya Kumar Jul 19 '13 at 6:22
    
As for me, you should write User-agent: Mediapartners-Google* Disallow: But it is not a thing. If I were in your shoes, I would check server logs and look for Goolgebot queries to /robots.txt –  Marian Jul 19 '13 at 6:44
    
Did you check encoding of your file? UTF-8 is recommended by Google. –  Zistoloen Jul 19 '13 at 7:54
    
@Marian: You should not include an asterisk on the end of the user agent (this is not a wildcard operator in this context - the lone * is a special case). John: Disallow: by itself (without a path element) does not disallow the robot - this directive will be ignored, so it will in fact allow the robot! –  w3d Jul 19 '13 at 11:56
add comment

3 Answers

up vote 4 down vote accepted

If you have a message saying that "Google couldn't crawl your site because we were unable to access the robots.txt file". Then it is not the contents of the robots.txt file that is in question, it is that Google simply couldn't access the file. And when Google can't access a robots.txt file then it won't crawl the site.

Using fetch as Googlebot in Webmaster Tools is a good idea. If your robots.txt file fetches successfully then it could be a past issue. If not, then you obviously need to look further to ensure Googlebot access.

share|improve this answer
add comment

There is no such official command as Allow in robots.txt. By default, everything is allowed. (However, it is possible to use Allow to give exceptions when you are disallowing many directory paths in one route. Often, there is no requirement for this though).

Not that I would expect it to cause an issue however.

There is no reason to specify the Mediapartners-Google user agent either as this too, is just saying allow the crawling of everything.

All your robots.txt needs from the above is the following:-

User-Agent: *
Disallow: /search/

User-agent: Mediapartners-Google
Disallow: /

Sitemap: http://latest-seo-news-updates.blogspot.com/feeds/posts/default?orderby=UPDATED

Google Webmaster Tools will report a warning to say X amount of URL's on your site were blocked by your robots.txt if you are disallowing bots to any part of your site, in which case you are at /search/. You can expand this notice to view specifically what URL's were blocked and you may well find that it is only the ones you want disallowed that Google Webmaster Tools is warning about.

You can also run an application such as Xenu to crawl your site and establish what URL's can be crawled specifically. You can also fetch as Googlebot and test your robots.txt file from within Google Webmaster Tools that will alert you to any further issues or at least complete details of any issues.

Edit: Upon further clarification, added Disallow directive for UA Mediapartners-Google.

share|improve this answer
    
Sorry, I deleted my comment! ... It would seem from the comments above that the OP does actually want to disallow the Mediapartners-Google robot, so the Disallow: directive (without a path element) in the question would seem to be incorrect. It should read Disallow: / (with a slash). –  w3d Jul 19 '13 at 12:02
    
Agreed - answer updated. Thanks for pointing out clarification as per comments. –  zigojacko Jul 19 '13 at 12:22
add comment
User-agent: Mediapartners-Google
Disallow: 
User-agent: *
Disallow: /search
Allow: /

The error comes up because of the Disallow: /search * means of engine bots by entering the comment /search, you are basically blocking them from going in the search index of your site. Take notice the code above:

User-agent: Mediapartners-Google
Disallow: 

Disallow: means that you are allowing the adsense bots to crawl everywhere unrestricted.

The Allow: / may not be interpreted by older bots but it well interpreted by google bot.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.