33

I have a link to PDF document on a public web-page. How do I prevent search engines from indexing this link and PDF document?

The only idea I thought of is to use CAPTCHA. However, I wonder if there are any magic words that tell a search engine to not index the link and PDF document? Options using PHP or JavaScript are also fine.

Just to make it clear. I do not want to encrypt PDF and protect it with password. I just want to make it invisible for search engines, but not for users.

Improve this question
2

5 Answers 5

Reset to default
37

To prevent your PDF file (or any non HTML file) from being listed in search results, the only way is to use the HTTP X-Robots-Tag response header, e.g.:

X-Robots-Tag: noindex

You can do this by adding the following snippet to the site's root .htaccess file or httpd.conf file:

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

Note that for the above to work, you must be able to modify the HTTP headers of the file in question. Thus you may not be able to do this, for example, on GitHub Pages.

Also note that robots.txt does not prevent your page from being listed in search results.

What it does is stop the bot from crawling your page, but if a third party links to your PDF file from their website, your page will still be listed.

If you stop the bot from crawling your page using robots.txt, it will not have the chance to see the X-Robots-Tag: noindex response tag. Therefore, never ever ever disallow a page in robots.txt if you employ the X-Robots-Tag header. More info can be found on Google Developers: Robots Meta Tag.

Improve this answer
4
  • I don't follow how robots.txt with an entry for the page would prevent the X-Robots-Tag to be respected/obeyed if the page with that X-Robots-Tag was requested in the future.
    – Don Cheadle
    Dec 22, 2016 at 4:56
  • 1
    I thought that when the page is requested, it("Google") would see the X-Robots-Tag header, and then know not to index it... but it looks like the X-Robots-Tag is only useful for crawlers which then know not to index it developers.google.com/webmasters/control-crawl-index/docs/…
    – Don Cheadle
    Dec 22, 2016 at 5:03
  • Since you are using the regex enabled Files directive, you should consider using instead FilesMatch instead, as suggested here stackoverflow.com/q/14792381/1262357
    – Gruber
    Jan 6, 2019 at 5:37
  • @Gruber Although there's no need to use "the regex enabled Files directive" here in the first place. The ordinary <Files> directive with a wildcard can be used to match all .pdf files (which seems to be the intention here). eg. <Files "*.pdf"> - which is arguably preferable in this situation.
    – MrWhite
    Dec 1, 2020 at 15:24
18

There are multiple ways to do this (combining them is obviously a sure way to accomplish this):

1) Use robots.txt to block the files from search engines crawlers:

User-agent: *
Disallow: /pdfs/ # Block the /pdfs/directory.
Disallow: *.pdf  # Block pdf files. Non-standard but works for major search engines.

2) Use rel="nofollow" on links to those PDFs

<a href="something.pdf" rel="nofollow">Download PDF</a>

3) Use the x-robots-tag: noindex HTTP header to prevent crawlers from indexing them. Place this code in your .htaccess file:

<FilesMatch "\.pdf$">
  header set x-robots-tag: noindex
</FilesMatch>
Improve this answer
1
  • 3
    Counterintuitively, using both x-robots-tag and robots.txt at the same time is not a good idea and may cause the content to be indexed anyways. If you use both robots.txt and x-robots-tag: noindex, the crawler will never crawl or see the x-robots-tag because it first honors the robots.txt. robots.txt does not prevent resources from being indexed, only from being crawled, so the best solution is to use the x-robots-tag header, yet allow the search engines to crawl and find that header by leaving your robots.txt alone.
    – Maximillian Laumeister ♦
    Dec 6, 2018 at 5:09
2

If you nginx powered development instances are showing up in Google search results, there is a quick and easy way to prevent search engines from crawling your site. Add the following line to the location block of your virtualhost configuration file for the block that you want to prevent crawling.

add_header  X-Robots-Tag "noindex, nofollow, nosnippet, noarchive";
Improve this answer
1

You can use robots.txt file. You can read more here.

Improve this answer
1
  • 4
    Using robots.txt does not prevent indexing. It only prevents crawling. See Pacerier's answer for correct solution.
    – toxalot
    Nov 25, 2013 at 15:48
1

Not sure if this sill might bring some value to anyone, but we have recently encountered a problem that our on-premise GSA box is unwilling to index PDF file.

Google Support worked with the issue and their response is that it is related to the fact that this PDF document has a custom property set (File -> Document Properties -> Custom (tab))

name: robots
value: noindex

which prevented it from being properly indexed by GSA.

If you have access to the document and can modify it's properties this might work ... at lease for GSA.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.