Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

In Google Webmaster Tools, I have 832 errors in the sitemap with the error being "URL restricted by robots.txt". I don't have a robots.txt file, and the robots meta tag equals "index, follow".

I did have a robots.txt file, and the date it existed could well correspond with the date of the crawl errors in Google Webmaster Tools.

However, the dashboard shows these crawl errors remain, "updated Sept 23, 2011" (yesterday). All of the crawl errors are about 3 weeks old, which is probably before I deleted robots.txt.

Is the fact the crawl errors remain as of yesterday indicating Google has not recrawled the site?

Google has tried to download /robots.txt again, as recently as 4 hours ago, and has not found it, and will not of for weeks.

Do I need to ping Google to crawl the site again?

share|improve this question

1 Answer 1

up vote 2 down vote accepted

You might consider putting up a more permissive robots.txt so that Googlebot finds something to download and work with. It will wait a while if robots.txt returns a 404 error, in case that was unintentional on your part.

Try:

User-Agent: *
Disallow:
share|improve this answer
    
Ahhh. Clever Google. Clever Mike Scott. Should I wait for Google's next scheduled check of robots.txt, or can I prompt it? Also, I had asked Google to remove /robots.txt, which was successful. Does Google continue to look for it if I have asked for it to be removed from the index? –  Steve Sep 24 '11 at 18:09

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.