I have a website which still shows old robots.txt in the google webmaster tools.

User-agent: *
Disallow: /

Which is blocking Googlebot. I have removed old file updated new robots.txt file with almost full access & uploaded it yesterday but it is still showing me the old version of robots.txt Latest updated copy contents are below

User-agent: *
Disallow: /flipbook/
Disallow: /SliderImage/
Disallow: /UserControls/
Disallow: /Scripts/
Disallow: /PDF/
Disallow: /dropdown/

I submitted request to remove this file using Google webmaster tools but my request was denied

I would appreciate if someone can tell me how i can clear it from the google cache and make google read the latest version of robots.txt file.

  • Hei…I didn’t know we could use * directive on Disallow. Nice guide by the way, I wanna post article about blocking content. Your post help me a lot.
    – user18544
    Commented Sep 20, 2012 at 18:25

3 Answers 3


This is from Google Webmaster Developers site https://developers.google.com/webmasters/control-crawl-index/docs/faq

How long will it take for changes in my robots.txt file to affect my search results?

First, the cache of the robots.txt file must be refreshed (we generally cache the contents for up to one day). Even after finding the change, crawling and indexing is a complicated process that can sometimes take quite some time for individual URLs, so it's impossible to give an exact timeline. Also, keep in mind that even if your robots.txt file is disallowing access to a URL, that URL may remain visible in search results despite that fact that we can't crawl it. If you wish to expedite removal of the pages you've blocked from Google, please submit a removal request via Google Webmaster Tools.

And here are specifications for robots.txt from Google https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

If your file's syntax is correct the best answer is just wait till Google updates your new robots file.


Usually Google checks regularly for changes to your robots.txt file.

If you sign up for a Google Webmaster Tools account you can find out more information, including the last time Google checked your robots.txt file.

EDIT: You can't request that your robots.txt file be removed either. The URL removal tool is for removing pages from the search index.

Have you confirmed that the new file has been uploaded correctly? There have been times I have tried to FTP a file to the server, but when I checked online the changes weren't showing up. When I logged in to my hosting control panel, and viewed my files, it was still the old file. Apparently the file wasn't transferred properly by my FTP client, although it wasn't displaying any errors.

So the first thing to do is confirm that your file was uploaded correctly bu going to the robots.txt file in your browser, and seeing what is there.

If the file was uploaded correctly, and you can see the new file in the browser, then your file is being cached somewhere else. Some CDN's cache static files, so this could be the cause if you use a CDN.

  • I do have GWM account and i have mentioned that Google Webmaster is still reading the old file not the new robots.tx file.
    – Learning
    Commented May 21, 2012 at 5:18

The first thing to do is to ensure that you correctly uploaded the new robots.txt file.

Visit www.yourwebsite.com/robots.txt from your browser to check it.

Not the answer you're looking for? Browse other questions tagged or ask your own question.