Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 1 down vote favorite

I've recently found that "Google can't find your site's robots.txt" in crawl errors. When I tried Fetching as Google, I got result "SUCCESS", then I tried looking at crawl errors and it still shows "Google can't find your site's robots.txt".

What can I do to resolve this issue? Before this issue arose, my site was indexed within a few mintues, but now I find that it took time to be indexed in Google's search.

When I access http://mydomain.com/robots.txt, it shows the data below:

User-agent:

*Disallow: /wp-admin/

Disallow: /wp-includes/

I found Blocked URLs = 0, also no any other errors.

Is there any other thing I need to change? Or what could be the solution for this? Any help would be appreciated.

share|improve this question
1  
Is this a recurring error in GWT? Check the date. I had the same error on 30th Sept on one of my sites. However, the same error displays whenever I go into GWT which makes it look as if it is persistent, but it is the same (old) message that is displayed. – w3d Oct 18 '12 at 16:45
1  
If you see it's an old message, simply go to Webmaster Tools → All Messages, find the offending message and tick the check box to the left of it to activate the delete button. – stealthyninja Oct 18 '12 at 17:01

2 Answers

up vote 1 down vote

Make sure that in Settings → Privacy in the back-end you've checked Allow search engines to index this site and not the other one which is Ask search engines not to index this site.

Also, that was probably just a typo, but your robots.txt should look like this:

User-agent: * 

Disallow: /wp-admin/
Disallow: /wp-includes/
share|improve this answer
Where is this Settings > Privacy option you mention? I do not see this in GWT? – w3d Oct 18 '12 at 16:41
1  
@w3d - It's not in GWT, it's in WordPress which is the software the OP is using for his site. – stealthyninja Oct 18 '12 at 16:53
up vote 0 down vote

Side note: Your robots.txt syntax is not correct:

User-agent:

*Disallow: /wp-admin/

Disallow: /wp-includes/
  • I guess the * moved to the next line accidently; it should be User-agent: *
  • *Disallow will be unknown to most parsers and therefor the followed rule will be ignored.
  • no line breaks allowed inside of a block

So it should be:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.