2

There are already a variety of posts on how to block certain files (in my case, PDFs) from a search engine like Google. The most relevant for this post was here: How to protect PDF file from indexing. However, in that post, the final answer was never quite clear. Based on these three sites:

I think I understand the recommendation. Essentially, we should not use robots.txt to disallow crawling/indexing of files. We should instead use X-Robots-Tag.

This brings me to three questions, which is really so I can be absolutely sure that what follows would work.

Question 1: Suppose I want to disallow search engine indexing to any files within a subfolder of my site, www.mysite.com/secret

I would create a .htaccess file in the subfolder with the following:

Header set X-Robots-Tag "noindex, nofollow"

Alternatively, if I wish to disallow access in the secret subfolder only to PDFs, I would use (again within a separate .htaccess in the subfolder):

<FilesMatch ".doc$">
Header set X-Robots-Tag "index, noarchive, nosnippet"
</FilesMatch>

Question 2: Is there any advantage to doing the same for the main .htaccess file in the website root directory? If so, how do you alter the above two statements for subdirectories? On Google's site they suggest:

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

Do I change it to "secret/\.pdf$"instead? I am unsure of forward vs. backward slashes.

Question 3: Suppose I have a separate PDF document on a separate page that links the PDF in the secret folder. Even with the .htaccess x-robots tag block in place, does the third party linking break the non-indexing command?

  • 1
    Do you want to disallow crawling or indexing? – unor Jul 29 '15 at 11:43
  • 1
    Do you want to disallow crawling or indexing? there's never any guarantee that Google or any other search bot won't crawl anything, even with blocks in robots, no index and no follow. – Simon Hayter Jul 29 '15 at 11:49
  • Out of interest why do you want to block PDF ? – Simon Hayter Jul 29 '15 at 11:49
  • 1
    Then <FilesMatch ".pdf$"> will stop Google indexing PDF's all together. You can reforce it with robots.txt if you wish but its not required unless your system goes AWOL and removes the x tag. – Simon Hayter Jul 29 '15 at 11:52
  • 1
    @SimonHayter You don't "reforce it with robots.txt" - if you block it with robots.txt then Google should never even make the request and so would never see the "noindex" header. – MrWhite Jul 30 '15 at 20:30
3

You have done your research and seem to have a good handle on the situation. To sum up:

Using robots.txt would prevent search engines from crawling the PDF files. If third party sites linked directly to the PDF files, then search engines might include the URLs in the search index (but would still not be able to index their contents.)

Using X-Robots-Tag "noindex, nofollow" will prevent search engines from indexing the PDF files even though they may crawl them. Third party sites directly linking to the files will still not cause the PDF files to get indexed.

You cannot use both methods. If you block the PDF files with robots.txt search engines will never see the header and may still index the URLs.


Your first FilesMatch matching looks correct if you substitute pdf for doc. The rule inside it looks like it would allow indexing, so you may have pasted in the wrong thing.

If you wanted to put it in the root directory you would need to use secret/.*\.pdf$ instead. The only advantage to doing so might be to centralize all your rules in one place.

  • Thank you. I forgot to ask: is there a way to check whether the X-Robots-Tag is working properly, or do you have to wait and find out whether it pops up in searches? – TSGM Jul 29 '15 at 12:02
  • Also, is it known whether indexing is enough to stop articles from showing up in Google Scholar? It does not seem to state here: scholar.google.co.uk/intl/en/scholar/inclusion.html Given what you have described, stopping Google from crawling is too difficult a task, so I would hope that Google Scholar will not include a page if there is a block to indexing. – TSGM Jul 29 '15 at 12:02
  • I use command line tools such as curl to see the headers: curl --head 'http:/example.com/secret/some.pdf' – Stephen Ostermiller Jul 29 '15 at 13:21
  • I've never worked with Google scholar. I'd recommend asking "Does Google Scholar respect a X-Robots-Tag "noindex, nofollow" header' as a separate question. – Stephen Ostermiller Jul 29 '15 at 13:22
  • Stopping Google from crawling with robots.txt is 99% effective. It only fails when a third party links to it, and even then it only shows up for keywords that are present in the URL or used in the anchor text of the links. I've used robots.txt in conjunction with file names that change periodically. That is very effective as well. – Stephen Ostermiller Jul 29 '15 at 13:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.