Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have link to PDF document on a public web-page. How to prevent search engines to index this link and PDF document. The only idea that came to my mind is to use CAPTCHA. However I wounder if there are any magic words that tell an engine do not index following link and PDF document? Any options using PHP or JavaScript are also fine.

Just to make it clear, I do not want to encrypt PDF and protect it with password. Just want to make it invisible for search engines, but not for users.

share|improve this question
    
Similar: webmasters.stackexchange.com/questions/14520/… but I've additionally asked how to allow search engines to read the PDF's for ranking of pages & site but not directly list them in the SERP. Relevant docs increase rankings –  AgA Sep 12 '13 at 3:59
    
@AgA the link you provided is a link to this question - creating a link loop. –  toxalot Nov 25 '13 at 15:51
add comment

migrated from stackoverflow.com May 28 '11 at 16:22

This question came from our site for professional and enthusiast programmers.

5 Answers

There are multiple ways to do this (combining them is obviously a sure way to accomplish this):

1) Use robots.txt to block the files from search engines crawlers:

User-agent: *
Disallow: /pdfs/ # Block the /pdfs/directory.
Disallow: *.pdf  # Block pdf files. Non-standard but works for major search engines.

2) Use rel="nofollow" on links to those PDFs

<a href="something.pdf" rel="nofollow">Download PDF</a>

3) Use the x-robots-tag: noindex HTTP header to prevent crawlers from indexing them. Place this code in your .htaccess file:

<FilesMatch "\.pdf$">
  header set x-robots-tag: noindex
</FilesMatch>
share|improve this answer
    
Never ever ever block a X-Robots-Tag page with robots.txt: webmasters.stackexchange.com/a/49124/7654 –  Pacerier Jun 7 '13 at 16:24
    
That's why I set it using Apache. It will always send the header when the PDF is requested. –  John Conde Jun 7 '13 at 16:35
1  
that's not what I'm saying.. I meant we should not block pdf files using robots.txt. Robots meta tags and X-Robots-Tag HTTP headers are discovered when a URL is crawled. If you block a page through robots.txt, then the spider would not have a chance to see the X-Robots-Tag. If the spider do not see the X-Robots-Tag, it will list the url in search results as long as there are external links to that URL. In other words, all files blocked by robots.txt are list-able in search results. See webmasters.stackexchange.com/a/49124/7654 –  Pacerier Jun 7 '13 at 17:35
add comment

To prevent your PDF file (or any non HTML file) from being listed in search results, the only way is to use the HTTP X-Robots-Tag response header, for example:

X-Robots-Tag: noindex

robots.txt does not prevent your page from being listed in search results. What it does is stop the bot from crawling your page, although if someone else links to your PDF file from their website, your page will still be listed.

If you stop the bot from crawling your page using robots.txt, it will never see the X-Robots-Tag: noindex response tag, thus never ever disallow a page in robots.txt if you employ the X-Robots-Tag header. This is furthur elaborated in Google Developers: Robots Meta Tag.

share|improve this answer
1  
Wish I could give you more votes. I don't have enough rep to down vote the others. –  toxalot Nov 25 '13 at 15:52
add comment

You can use robots.txt file. You can read more here.

share|improve this answer
    
Using robots.txt does not prevent indexing. It only prevents crawling. See Pacerier's answer for correct solution. –  toxalot Nov 25 '13 at 15:48
add comment

You can use a robots.txt file. Search engines that honour that file will not index the PDF. Just use the command for disallowing the indexing of a file and designate which folder or PDF file you don't want the search engines to index.

share|improve this answer
    
Using robots.txt does not prevent indexing. It only prevents crawling. See Pacerier's answer for correct solution. I did read recently about using a noindex directive (rather than disallow) inside the robots.txt file, but it's undocumented and non-standard. And now I can't find where I read it. –  toxalot Nov 25 '13 at 15:57
add comment

Maybe use robots.txt to disallow index?

share|improve this answer
    
Using robots.txt does not prevent indexing. It only prevents crawling. See Pacerier's answer for correct solution. –  toxalot Nov 25 '13 at 15:58
    
Typically Google will not index files that are blocked by robots.txt unless there are several external links to them. Unless the PDF files are externally linked, robots.txt would be a reasonable solution. –  Stephen Ostermiller Nov 30 '13 at 22:32
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.