1

In Google Webmaster Tools, I have 832 errors in the sitemap with the error being "URL restricted by robots.txt". I don't have a robots.txt file, and the robots meta tag equals "index, follow".

I did have a robots.txt file, and the date it existed could well correspond with the date of the crawl errors in Google Webmaster Tools.

However, the dashboard shows these crawl errors remain, "updated Sept 23, 2011" (yesterday). All of the crawl errors are about 3 weeks old, which is probably before I deleted robots.txt.

Is the fact the crawl errors remain as of yesterday indicating Google has not recrawled the site?

Google has tried to download /robots.txt again, as recently as 4 hours ago, and has not found it, and will not of for weeks.

Do I need to ping Google to crawl the site again?

|improve this question
2

You might consider putting up a more permissive robots.txt so that Googlebot finds something to download and work with. It will wait a while if robots.txt returns a 404 error, in case that was unintentional on your part.

Try:

User-Agent: *
Disallow:
|improve this answer
  • Ahhh. Clever Google. Clever Mike Scott. Should I wait for Google's next scheduled check of robots.txt, or can I prompt it? Also, I had asked Google to remove /robots.txt, which was successful. Does Google continue to look for it if I have asked for it to be removed from the index? – Steve Sep 24 '11 at 18:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.