Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I've recently found that "Google can't find your site's robots.txt" in crawl errors. When I tried Fetching as Google, I got result "SUCCESS", then I tried looking at crawl errors and it still shows "Google can't find your site's robots.txt".

What can I do to resolve this issue? Before this issue arose, my site was indexed within a few mintues, but now I find that it took time to be indexed in Google's search.

When I access http://mydomain.com/robots.txt, it shows the data below:

User-agent:

*Disallow: /wp-admin/

Disallow: /wp-includes/

I found Blocked URLs = 0, also no any other errors.

Is there any other thing I need to change? Or what could be the solution for this? Any help would be appreciated.

share|improve this question
1  
Is this a recurring error in GWT? Check the date. I had the same error on 30th Sept on one of my sites. However, the same error displays whenever I go into GWT which makes it look as if it is persistent, but it is the same (old) message that is displayed. –  w3d Oct 18 '12 at 16:45
1  
If you see it's an old message, simply go to Webmaster Tools → All Messages, find the offending message and tick the check box to the left of it to activate the delete button. –  stealthyninja Oct 18 '12 at 17:01

3 Answers 3

Make sure that in Settings → Privacy in the back-end you've checked Allow search engines to index this site and not the other one which is Ask search engines not to index this site.

Also, that was probably just a typo, but your robots.txt should look like this:

User-agent: * 

Disallow: /wp-admin/
Disallow: /wp-includes/
share|improve this answer
    
Where is this Settings > Privacy option you mention? I do not see this in GWT? –  w3d Oct 18 '12 at 16:41
1  
@w3d - It's not in GWT, it's in WordPress which is the software the OP is using for his site. –  stealthyninja Oct 18 '12 at 16:53

Side note: Your robots.txt syntax is not correct:

User-agent:

*Disallow: /wp-admin/

Disallow: /wp-includes/
  • I guess the * moved to the next line accidently; it should be User-agent: *
  • *Disallow will be unknown to most parsers and therefor the followed rule will be ignored.
  • no line breaks allowed inside of a block

So it should be:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
share|improve this answer

If you have a lot of websites stored on the same server, and use the same Google Analytics account for them all, and also use a third party hosting service such as GoDaddy, NetWork Solutions, etc... with a dedicated server, the hosting company may be blocking the Googlebot after it tries a certain number of the websites that you have hosted.

The Googlebot and the Fetch as Google tool run on two different IP numbers, which would account for the ability for you to Fetch as Google successfully, but that the Googlebot could not find your robots.txt.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.