Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have updated some url's and files in robots.txt file to block url's and files from google search results but, still files displaying in search results. As per a suggestion from a site I tried to update the robots.txt by below steps.

  1. In Google Webmaster tools, Health -> Fetch as Google -> type the url and click the fetch button.

but, still files displaying in search results.

Note: in Google Webmaster tools, Health -> Blocked URL's -> robots.txt file - downloaded date looks two dates back.

share|improve this question

migrated from serverfault.com Oct 9 '12 at 15:43

This question came from our site for professional system and network administrators.

4 Answers 4

up vote 11 down vote accepted

It takes some time for Googlebot to crawl your site. This amount of time can vary depending on the number of links to your site, among other factors. After your site is crawled, the URLs in the robot.txt file will be updated.

share|improve this answer
2  
+1 This usually takes hours to days, but could be longer too. –  Chris S Oct 9 '12 at 15:43

May be that link is in indexed database of Google. Don't worry when it has been removed from Google database it will not show in SERPs.

share|improve this answer

Robots.txt can only prevent a search engine from crawling, it won't prevent indexing if the page or file has links from pages, internal or external to your site, which are crawlable.

To be absolutely sure that a page won't be indexed, apply <meta name="robots" content="noindex">. For any non-HTML content, like images of PDFs, use the equivalent HTTP header: X-Robots-Tag: noindex.

If you use these methods, it's unnecessary to submit requests for Google to remove them for you.

More discussion of both from Google here.

share|improve this answer
    
+1 for great info. To those who have a template based site design and not static html pages, they should take care they do not deindex their entire site by accident. So some programming to select pages my be required going this route. Also for popular CMS like wordpress and drupal for example, there exists plugins/extensions to achieve what GDav suggested as well. –  Anthony Hatzopoulos Oct 11 '12 at 13:48
    
Hi GDav Thanks for your response. if the non-HTML content is not editable (like ebook- pdf format)then how can we set the header? –  Haridharan Oct 16 '12 at 5:59
    
@Haridharan bit.ly/Xh6vw2 –  GDav Oct 16 '12 at 7:45

robots.txt only specifies how Google should or should not crawl your site and as Mike said if you solely rely on this vector it will take a long time to have the pages removed from the index.

So you will also need to make a removal request if you want it to happen faster and have Google remove the pages from the index. When you are in Webmaster Tools go to Optimization > Remove URLs and follow the instructions.

enter image description here

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.