3

I've recently found that "Google can't find your site's robots.txt" in crawl errors. When I tried Fetching as Google, I got result "SUCCESS", then I tried looking at crawl errors and it still shows "Google can't find your site's robots.txt".

What can I do to resolve this issue? Before this issue arose, my site was indexed within a few mintues, but now I find that it took time to be indexed in Google's search.

When I access http://mydomain.com/robots.txt, it shows the data below:

User-agent:

*Disallow: /wp-admin/

Disallow: /wp-includes/

I found Blocked URLs = 0, also no any other errors.

Is there any other thing I need to change? Or what could be the solution for this? Any help would be appreciated.

2
  • 1
    Is this a recurring error in GWT? Check the date. I had the same error on 30th Sept on one of my sites. However, the same error displays whenever I go into GWT which makes it look as if it is persistent, but it is the same (old) message that is displayed.
    – MrWhite
    Oct 18, 2012 at 16:45
  • 1
    If you see it's an old message, simply go to Webmaster Tools → All Messages, find the offending message and tick the check box to the left of it to activate the delete button. Oct 18, 2012 at 17:01

3 Answers 3

2

Make sure that in Settings → Privacy in the back-end you've checked Allow search engines to index this site and not the other one which is Ask search engines not to index this site.

Also, that was probably just a typo, but your robots.txt should look like this:

User-agent: * 

Disallow: /wp-admin/
Disallow: /wp-includes/
2
  • Where is this Settings > Privacy option you mention? I do not see this in GWT?
    – MrWhite
    Oct 18, 2012 at 16:41
  • 1
    @w3d - It's not in GWT, it's in WordPress which is the software the OP is using for his site. Oct 18, 2012 at 16:53
0

Side note: Your robots.txt syntax is not correct:

User-agent:

*Disallow: /wp-admin/

Disallow: /wp-includes/
  • I guess the * moved to the next line accidently; it should be User-agent: *
  • *Disallow will be unknown to most parsers and therefor the followed rule will be ignored.
  • no line breaks allowed inside of a block

So it should be:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
0

If you have a lot of websites stored on the same server, and use the same Google Analytics account for them all, and also use a third party hosting service such as GoDaddy, NetWork Solutions, etc... with a dedicated server, the hosting company may be blocking the Googlebot after it tries a certain number of the websites that you have hosted.

The Googlebot and the Fetch as Google tool run on two different IP numbers, which would account for the ability for you to Fetch as Google successfully, but that the Googlebot could not find your robots.txt.

Not the answer you're looking for? Browse other questions tagged or ask your own question.