A few weeks ago we have figured out that Google has indexed some information we would rather keep in some confidentiality, in the format of individual PDF files. Our assumption was that this was a problem with our robots.txt we had overlooked. Even though we are not sure whether or not this is the case, we are certain that the robots.txt file is in a valid format and is, according to Google's webmaster tools, blocking the files.

However, even after this adjustment that has been made weeks ago, Google still has the PDF files indexed, but does tell us further information cannot be provided due to the robots.txt file being present. As you can hopefully understand, this is unwanted behaviour due to the nature of the documents. I am aware that there is a request page being provided by Google for this purpose, but there are a lot of files. Is there an easier way to get Google to remove all of the files from its search engine? If not, is there anything else you could advise us to do besides manually requesting Google to remove every single page?

  • It can take weeks to months depending on how many PDFS you have, Google doesn't process all the URLS at once. Jun 10, 2014 at 9:14
  • Your only options are to manually remove each URL via Google WMT or wait for Google to eventually drop the URL's from their index. Providing your blocking the crawler from accessing the content of those pages (subsequently the URL's won't rank for much), waiting for them to drop out of the index naturally should suffice?
    – zigojacko
    Jun 10, 2014 at 9:25
  • 1
    (robots.txt blocks crawling, not indexing) If the pdfs are in their own directory/folder then you could submit a directory removal via webmaster tools.
    – user29671
    Jun 10, 2014 at 11:21

3 Answers 3


As mentioned in the comments, robots.txt only blocks crawling of files, it wont remove them from the SERPs.

To remove PDF files from the SERPs add an X-Robots-Tag: noindex in the HTTP header used to serve the file. They’ll drop out over time if you use the X-Robot-Tag with the noindex directive, or as mentioned you can speed up the removal process in GWT. Although if removing in GWT, you dont need to add the X-Robots-Tag if they are already blocked in robots.txt


You can include meta tag namely no index, no follow or else you can exclude with disallow with the robots. txt.


I think this article would be useful for you.

  • Could you please explain a little bit more what is this about to help the OP?
    – Zistoloen
    Jun 10, 2014 at 12:59
  • 1
    While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. Jun 11, 2014 at 0:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.