I am trying to include the following in my response header to all resources in multiple directories (dirone/dirtwo/dirthree/direfour) from an Apache server.

X-Robots-Tag: noindex, nofollow

In my httpd.conf file I have added the setting to the dirone underneath my main directory declaration.

It is not producing the additional item in the response header. Do you see what I am doing wrong?

<Directory />
    Options FollowSymLinks Includes
    AllowOverride None

<Directory "/web/sitename/docs/dirone">
    Header set X-Robots-Tag "noindex, nofollow"

Additional information. I am moving away from the robots.txt approach, we do not use .htaccess files and I want the header in all resources in those directories and their sub-directories to have X-Robots-Tag: noindex, nofollow in them as well.

Thanks for your help.

  • Just to confirm, is /web/sitename/docs/dirone a physical directory (or symlink) on the filesystem? – MrWhite Dec 14 '15 at 1:17
  • It is the physical directory – wall-nut Dec 14 '15 at 4:45
  • 1
    The syntax above did work once the /dev/ was added to the path. Thanks all. – wall-nut Dec 14 '15 at 15:33

I think the problem is that the Header directive shouldn't go in a Directory section. Instead it should go with the other configuration for sitename and use a Files directive.

So in your sitename configuration (even in your .htaccess) you should use:

<Files "dirone/*">
    Header set X-Robots-Tag "noindex, nofollow"
  • The Header directive is permitted in <Directory> sections and this can certainly work. So it would seem there is something else going on here? – MrWhite Dec 14 '15 at 1:17
  • @w3d Right, something else must be going on. – Stephen Ostermiller Dec 14 '15 at 2:18
  • 1
    Thanks. it was something else. The path I used was incorrect. – wall-nut Dec 14 '15 at 15:35
  • @wall-nut I know it's an old question, but look at my answer and see if it's ok for you. – jrosell Sep 2 '19 at 15:12

Try with the following:

SetEnvIf Request_URI ^/dirone.* NOINDEXNOFOLLOW
SetEnvIf Request_URI ^/dirtwo.* NOINDEXNOFOLLOW
SetEnvIf Request_URI ^/dirthree.* NOINDEXNOFOLLOW
SetEnvIf Request_URI ^/direfour.* NOINDEXNOFOLLOW
Header set X-Robots-Tag "noindex, nofollow" env=NOINDEXNOFOLLOW

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.