I added a robots.txt file to one of my sites a week ago, which should have prevented Googlebot from attempting to fetch certain URLs. However, this weekend I can see Googlebot loading those exact URLs.

Does Google cache robots.txt and, if so, should it?

7 Answers 7


I would strongly recommend registering your site with Google Search Console (previously Google Webmaster Tools). There is a crawler access section under site configuration that will tell you when your robots.txt was last downloaded. The tool also provides a lot of detail as to how the crawlers are seeing your site, what is blocked or not working, and where you are appearing in queries on Google.

From what I can tell, Google downloads the robots.txt often. The Google Search Console site will also let you specifically remove URLs from the index, so you can remove those ones you are now blocking.

  • 2
    I checked webmaster tools: the robots.txt file is valid and it was most recently fetched 17 hours before the most recent visit to those pages by googlebot. I suspect it is a question of propagation through google's network - eventually all googlebot servers will catch up with the robots.txt instructions.
    – DDM
    Commented Aug 16, 2010 at 14:31
  • The Google bot is not using the robots.txt as often as updates are reported in the Search Console. It is has been four weeks since I made an update, and the Google bot still uses a bad robots.txt - and it destroys our traffic and rankings. Commented Apr 29, 2019 at 13:40

Persevere. I changed from robots.txt towards meta noindex,nofollow. In order to make the meta work the blocked addresses in robots.txt had to first be unblocked.

I did this brutally by deleting the robots.txt altogether (and delcaring it in google's webmaster).

The robots.txt removal process as seen in the webmaster tool (number of pages blocked) took 10 weeks to be completed, of which the bulk was only removed by google during the last 2 weeks.

  • I tend to agree with you. We made a mistake and wrongly updated the robots.txt file. Google cached it, and it is using it four weeks after we corrected the mistake, and replaced it with a new robots.txt. I even manually submitted a refresh request in Google Webmaster Tools and... nothing. This is really bad as it resulted in lost traffic and rankings. :( Commented Apr 29, 2019 at 13:38

Yes, Google will obviously cache robots.txt to an extent - it won't download it every time it wants to look at a page. How long it caches it for, I don't know. However, if you have a long Expires header set, Googlebot may leave it much longer to check the file.

Another problem could be a misconfigured file. In the Webmaster Tools that danivovich suggests, there is a robots.txt checker. It will tell you which types of pages are blocked and which are fine.


Google's Documentation states that they will usually cache robots.txt for a day, but might use it for longer in if they get errors when trying to refresh it.

A robots.txt request is generally cached for up to one day, but may be cached longer in situations where refreshing the cached version is not possible (for example, due to timeouts or 5xx errors). The cached response may be shared by different crawlers. Google may increase or decrease the cache lifetime based on max-age Cache-Control HTTP headers.


Yes. They say they typically update it once a day, but some have suggested they may also check it after a certain number of page hits (100?) so busier sites are checked more often.

See https://webmasters.stackexchange.com/a/29946 and the video that @DisgruntedGoat shared above http://youtube.com/watch?v=I2giR-WKUfY.


From what I can see on the user accessible cache they do, what you need to do is type the URL of your robots.txt file into a Google Search and then click the little green dropdown arrow and click 'cached' (see image below) this will give you the the latest version of that page from Googles servers.

enter image description here


You can request its removal using Google's URL removal tool.

  • This does not answer the question.
    – MrWhite
    Commented Nov 15, 2015 at 21:15
  • why not the answer ? Commented Nov 16, 2015 at 1:26
  • Because the question is specifically about robots.txt, caching and crawling of URLs. One of the results of this might be that URLs aren't indexed, but that is not the question. (Google's URL removal tool is also only a "tempoary" fix, there are other steps you need to do to make it permanent.)
    – MrWhite
    Commented Nov 16, 2015 at 11:06

Not the answer you're looking for? Browse other questions tagged or ask your own question.