Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I added a robots.txt file to one of my sites a week ago, which should have prevented googlebot from attempting to fetch certain URLs. However, this weekend I can see googlebot loading those exact URLs.

Does Google cache robots.txt and, if so, should it?

share|improve this question
add comment

3 Answers

Yes. They say they typically update it once a day, but some have suggested they may also check it after a certain number of page hits (100?) so busier sites are checked more often.

See http://webmasters.stackexchange.com/a/29946 and the video that @DisgruntedGoat shared above http://youtube.com/watch?v=I2giR-WKUfY.

share|improve this answer
add comment

Yes, Google will obviously cache robots.txt to an extent - it won't download it every time it wants to look at a page. How long it caches it for, I don't know. However, if you have a long Expires header set, Googlebot may leave it much longer to check the file.

Another problem could be a misconfigured file. In the webmaster tools that danivovich suggests, there is a robots.txt checker. It will tell you which types of pages are blocked and which are fine.

share|improve this answer
    
See comment on this answer webmasters.stackexchange.com/questions/2272/… –  Quog Aug 16 '10 at 14:32
1  
@Quog: See this recent video: youtube.com/watch?v=I2giR-WKUfY Matt Cutts suggests that robots.txt is downloaded either once a day or about every 100 requests. –  DisgruntledGoat Aug 17 '10 at 10:36
add comment

I would strongly recommend registering your site with webmaster tools. There is a crawler access section under site configuration that will tell you when your robots.txt was last downloaded. The tool also provides a lot of detail as to how the crawlers are seeing your site, what is blocked or not working, and where you are appearing in queries on Google.

From what I can tell, google downloads the robots.txt often. The webmaster tools site will also let you specifically remove URLs from the index, so you can remove those ones you are now blocking.

share|improve this answer
2  
I checked webmaster tools: the robots.txt file is valid and it was most recently fetched 17 hours before the most recent visit to those pages by googlebot. I suspect it is a question of propagation through google's network - eventually all googlebot servers will catch up with the robots.txt instructions. –  Quog Aug 16 '10 at 14:31
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.