3

I am trying to analyse the .htaccess file of a website, specifically the following piece of code:

<FilesMatch ".pdf$">
<IfModule mod_headers.c>
Header set X-Robots-Tag "noindex, noarchive, nosnippet"
</IfModule>
</FilesMatch>

It is supposed to noindex all the PDF files of the website.

However, the PDF files are still there up and running, and I am sure about it because:

  • they still appear in SERPs
  • they are green in search console
  • the header check does not show any command about noindexing

How is that possible? My hypothesis is that there is some conflict in the .htaccess code.

The robots.txt file:

User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /web_service/
Disallow: /wp-admin/
Disallow: /xmlrpc.php

Sitemap: https://www.example.com/sitemap.xml
  • Do you have a robots.txt file? Could it be disallowing crawling of those files? Google can't see the headers if it can't crawl. – Stephen Ostermiller Oct 31 at 11:28
  • Hi @StephenOstermiller the robots.txt is file is as simple as possible, on purpose, and is not disallowing any PDF. If you tell me I can link it, I will. Thank you – FabrizioM Oct 31 at 12:01
  • You can link. Then we can check to see if your headers are actually getting set. – Stephen Ostermiller Oct 31 at 13:25
  • 2
    Your header isn't making it to your documents. I tried curl --head 'https://www.aurigaspa.com/wp-content/uploads/Top-Banking-Trends-for-2019-and-How-Banks-Need-to-Prepare-White-Paper-Auriga.pdf' – Stephen Ostermiller Oct 31 at 14:03
  • 2
    The directives you posted should ordinarily set the required header providing the .pdf file actually exists and the request is not internally rewritten to a script that manages the PDF response? The <IfModule> wrapper is not required and should be removed (at least during testing). The FilesMatch regex should strictly be "\.pdf$" to match a literal dot and not anything. It's possible there is a conflict, although it's difficult to imagine what "conflict" that might be? Where is this .htaccess located? Is the .htaccess file being processed? Do you have other .htaccess files? – MrWhite Oct 31 at 15:35
1

Your header isn't making it to your documents. I tried curl --head 'https://www.aurigaspa.com/wp-content/uploads/Top-Banking-Trends-for-2019-and-How-Banks-Need-to-Prepare-White-Paper-Auriga.pdf' curl is a command line program. You can run it yourself after SSHing into your server, or you can download and install it onto your home computer.

My guess is that mod_headers is either not installed or not enabled. You have the condition <IfModule mod_headers.c> in your rules. That will cause your directive to silently not work if mod_headers isn't installed or enabled.

I would suggest changing your .htaccess to:

<FilesMatch ".pdf$">
Header set X-Robots-Tag "noindex, noarchive, nosnippet"
</FilesMatch>

That could make you website unavailable. If that happens, you know you need to install or enable mod_headers. See How to install mod_headers on ServerFault.

  • THank you Stephen, I guess this was the problem indeed. Problem solved! – FabrizioM 2 days ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.