Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 1 down vote favorite

I have a website: http://46.105.30.3/

And I'm trying to get testing it using the Fetch as Googlebot option in the Google Webmasters Tools.

When I try it it returns Denied by robots.txt.

My robots.txt contains:

User-agent: *
Disallow: /js/
Disallow: /style/

Any ideas why the Googlebot is denied access?

share|improve this question
regarding "You can see your robots.txt there and last date googlebot grab it. You can test your robots file there too." I have same issue as noname above - blocked before - left it too long (forgot it to be honest), removed it in panik today went and told GWMT all about it, made that test, sent googlebot to fetch but still saying Denied by robots.txt!?!? You would think that a "fetch" I initiate would be in real-time and go cache the new robots.txt with a clear Allow: / and nothing else, but it still comes back thinking the old one is still the command it should listen to!?!? How can that be e – user12224 Dec 21 '11 at 7:52

1 Answer

up vote 3 down vote accepted

Your robots.txt looks good. Did you have an old one which blocked robots? Did you unblocked them recently? Maybe GWMT have the old one in its cache and you should wait a little bit before it gets refreshed. Sign into GWMT goto Site Configuration -> Crawler access. You can see your robots.txt there and last date googlebot grab it. You can test your robots file there too.

share|improve this answer
You're absolutely right. I used to have a robots.txt which blocked all agents. It is working fine now. – user6669 Apr 17 '11 at 11:08
One way around this is to go into Webmaster Tools down in diagnostics. Submit your robots.txt to the "Fetch as Googlebot" tool. Once it returns with a successful read of the robots.txt file, submit the file to Google. This has worked well to get over having Google baulking because of a previous "Deny All" robots.txt file getting stuck in Google's system. – Fiasco Labs Dec 23 '11 at 3:15

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.