There are already a variety of posts on how to block certain files (in my case, PDFs) from a search engine like Google. The most relevant for this post was here: How to protect PDF file from indexing. However, in that post, the final answer was never quite clear. Based on these three sites:
- Playing with the X-Robots-Tag
- Preventing your site from being indexed, the right way
- Google Developers Robots Meta Tag
I think I understand the recommendation. Essentially, we should not use robots.txt to disallow crawling/indexing of files. We should instead use X-Robots-Tag.
This brings me to three questions, which is really so I can be absolutely sure that what follows would work.
Question 1: Suppose I want to disallow search engine indexing to any files within a subfolder of my site, www.mysite.com/secret
I would create a .htaccess file in the subfolder with the following:
Header set X-Robots-Tag "noindex, nofollow"
Alternatively, if I wish to disallow access in the secret subfolder only to PDFs, I would use (again within a separate .htaccess in the subfolder):
<FilesMatch ".doc$">
Header set X-Robots-Tag "index, noarchive, nosnippet"
</FilesMatch>
Question 2: Is there any advantage to doing the same for the main .htaccess file in the website root directory? If so, how do you alter the above two statements for subdirectories? On Google's site they suggest:
<Files ~ "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</Files>
Do I change it to "secret/\.pdf$"
instead? I am unsure of forward vs. backward slashes.
Question 3: Suppose I have a separate PDF document on a separate page that links the PDF in the secret folder. Even with the .htaccess x-robots tag block in place, does the third party linking break the non-indexing command?
Do you want to disallow crawling or indexing?
there's never any guarantee that Google or any other search bot won't crawl anything, even with blocks in robots, no index and no follow.<FilesMatch ".pdf$">
will stop Google indexing PDF's all together. You can reforce it with robots.txt if you wish but its not required unless your system goes AWOL and removes the x tag.