I have updated some url's and files in robots.txt file to block url's and files from google search results but, still files displaying in search results. As per a suggestion from a site I tried to update the robots.txt by below steps.

  1. In Google Webmaster tools, Health -> Fetch as Google -> type the url and click the fetch button.

but, still files displaying in search results.

Note: in Google Webmaster tools, Health -> Blocked URL's -> robots.txt file - downloaded date looks two dates back.

4 Answers 4


It takes some time for Googlebot to crawl your site. This amount of time can vary depending on the number of links to your site, among other factors. After your site is crawled, the URLs in the robot.txt file will be updated.

  • 2
    +1 This usually takes hours to days, but could be longer too.
    – Chris S
    Commented Oct 9, 2012 at 15:43

robots.txt only specifies how Google should or should not crawl your site and as Mike said if you solely rely on this vector it will take a long time to have the pages removed from the index.

So you will also need to make a removal request if you want it to happen faster and have Google remove the pages from the index. When you are in Webmaster Tools go to Optimization > Remove URLs and follow the instructions.

enter image description here


Robots.txt can only prevent a search engine from crawling, it won't prevent indexing if the page or file has links from pages, internal or external to your site, which are crawlable.

To be absolutely sure that a page won't be indexed, apply <meta name="robots" content="noindex">. For any non-HTML content, like images of PDFs, use the equivalent HTTP header: X-Robots-Tag: noindex.

If you use these methods, it's unnecessary to submit requests for Google to remove them for you.

More discussion of both from Google here.

  • +1 for great info. To those who have a template based site design and not static html pages, they should take care they do not deindex their entire site by accident. So some programming to select pages my be required going this route. Also for popular CMS like wordpress and drupal for example, there exists plugins/extensions to achieve what GDav suggested as well. Commented Oct 11, 2012 at 13:48
  • Hi GDav Thanks for your response. if the non-HTML content is not editable (like ebook- pdf format)then how can we set the header?
    – Haridharan
    Commented Oct 16, 2012 at 5:59
  • @Haridharan bit.ly/Xh6vw2
    – GDVS
    Commented Oct 16, 2012 at 7:45

May be that link is in indexed database of Google. Don't worry when it has been removed from Google database it will not show in SERPs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.