Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

A few weeks ago we have figured out that Google has indexed some information we would rather keep in some confidentiality, in the format of individual PDF files. Our assumption was that this was a problem with our robots.txt we had overlooked. Even though we are not sure whether or not this is the case, we are certain that the robots.txt file is in a valid format and is, according to Google's webmaster tools, blocking the files.

However, even after this adjustment that has been made weeks ago, Google still has the PDF files indexed, but does tell us further information cannot be provided due to the robots.txt file being present. As you can hopefully understand, this is unwanted behaviour due to the nature of the documents. I am aware that there is a request page being provided by Google for this purpose, but there are a lot of files. Is there an easier way to get Google to remove all of the files from its search engine? If not, is there anything else you could advise us to do besides manually requesting Google to remove every single page?

share|improve this question
It can take weeks to months depending on how many PDFS you have, Google doesn't process all the URLS at once. – Simon Hayter Jun 10 '14 at 9:14
Your only options are to manually remove each URL via Google WMT or wait for Google to eventually drop the URL's from their index. Providing your blocking the crawler from accessing the content of those pages (subsequently the URL's won't rank for much), waiting for them to drop out of the index naturally should suffice? – zigojacko Jun 10 '14 at 9:25
(robots.txt blocks crawling, not indexing) If the pdfs are in their own directory/folder then you could submit a directory removal via webmaster tools. – user29671 Jun 10 '14 at 11:21

As mentioned in the comments, robots.txt only blocks crawling of files, it wont remove them from the SERPs.

To remove PDF files from the SERPs add an X-Robots-Tag: noindex in the HTTP header used to serve the file. They’ll drop out over time if you use the X-Robot-Tag with the noindex directive, or as mentioned you can speed up the removal process in GWT. Although if removing in GWT, you dont need to add the X-Robots-Tag if they are already blocked in robots.txt

share|improve this answer

You can include meta tag namely no index, no follow or else you can exclude with disallow with the robots. txt.

share|improve this answer

I think this article would be useful for you.

share|improve this answer
Could you please explain a little bit more what is this about to help the OP? – Zistoloen Jun 10 '14 at 12:59
While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. – Stephen Ostermiller Jun 11 '14 at 0:42

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.