Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

This question already has an answer here:

I have a website which still shows old robots.txt in the google webmaster tools.

User-agent: *
Disallow: /

Which is blocking Googlebot. I have removed old file updated new robots.txt file with almost full access & uploaded it yesterday but it is still showing me the old version of robots.txt Latest updated copy contents are below

User-agent: *
Disallow: /flipbook/
Disallow: /SliderImage/
Disallow: /UserControls/
Disallow: /Scripts/
Disallow: /PDF/
Disallow: /dropdown/

I submitted request to remove this file using Google webmaster tools but my request was denied

I would appreciate if someone can tell me how i can clear it from the google cache and make google read the latest version of robots.txt file.

share|improve this question

marked as duplicate by John Conde Jul 8 at 17:27

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

    
Hei…I didn’t know we could use * directive on Disallow. Nice guide by the way, I wanna post article about blocking content. Your post help me a lot. –  user18544 Sep 20 '12 at 18:25
add comment

3 Answers 3

This is from Google Webmaster Developers site https://developers.google.com/webmasters/control-crawl-index/docs/faq

How long will it take for changes in my robots.txt file to affect my search results?

First, the cache of the robots.txt file must be refreshed (we generally cache the contents for up to one day). Even after finding the change, crawling and indexing is a complicated process that can sometimes take quite some time for individual URLs, so it's impossible to give an exact timeline. Also, keep in mind that even if your robots.txt file is disallowing access to a URL, that URL may remain visible in search results despite that fact that we can't crawl it. If you wish to expedite removal of the pages you've blocked from Google, please submit a removal request via Google Webmaster Tools.

And here are specifications for robots.txt from Google https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

If your file's syntax is correct the best answer is just wait till Google updates your new robots file.

share|improve this answer
add comment

Usually Google checks regularly for changes to your robots.txt file.

If you sign up for a Google Webmaster Tools account you can find out more information, including the last time Google checked your robots.txt file.

EDIT: You can't request that your robots.txt file be removed either. The URL removal tool is for removing pages from the search index.

Have you confirmed that the new file has been uploaded correctly? There have been times I have tried to FTP a file to the server, but when I checked online the changes weren't showing up. When I logged in to my hosting control panel, and viewed my files, it was still the old file. Apparently the file wasn't transferred properly by my FTP client, although it wasn't displaying any errors.

So the first thing to do is confirm that your file was uploaded correctly bu going to the robots.txt file in your browser, and seeing what is there.

If the file was uploaded correctly, and you can see the new file in the browser, then your file is being cached somewhere else. Some CDN's cache static files, so this could be the cause if you use a CDN.

share|improve this answer
    
I do have GWM account and i have mentioned that Google Webmaster is still reading the old file not the new robots.tx file. –  KnowledgeSeeker May 21 '12 at 5:18
add comment

The first thing to do is to ensure that you correctly uploaded the new robots.txt file.

Visit www.yourwebsite.com/robots.txt from your browser to check it.

share|improve this answer
add comment

protected by John Conde Jul 8 at 17:27

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.