4

I have a directory full of .txt files that are used as part of a larger javascript application. The text files are not inteded to be directly accessed, and are used as includes on separate pages as part of the application functionality).

So.. I added the directory to my robots.txt file because I don't want the files to show up independently in search results.

However, robots.txt simply makes the .txt files inaccessible to robots. It doesn't actually make the .txt files un-indexable.

What I really want to do is "noindex" the .txt files. But .txt files aren't like .html files.. a noindex meta tag will be visible in the text.

So how does one "noindex" a file that's raw text without corrupting the text with a tag?

Is there some other way to noindex a text file, externally? Or noindex the overall directory?

7

The solution is the same as for X-Robots noindex a folder of PDFs and Prevent XML sitemaps from showing up in Google search results. You use the X-Robots-Tag HTTP header rather than a meta tag. The HTTP header served with the txt file should look like:

X-Robots-Tag: noindex

After implementing the HTTP header, remove the disallow from your robots.txt. In order for crawlers to be able to see and honor the header, you need to make the txt files crawlable.


Under Apache web servers (most shared hosting), put the following code in the .htaccess file in the directory containing the .txt files:

<FilesMatch ".txt$">
  Header set X-Robots-Tag "noindex"
</FilesMatch>

If you are using nginx, your configuration would need to contain (source):

location ~* ^/txtfiledir/.*\.txt$ {
  add_header  X-Robots-Tag "noindex";
}

If you are using IIS as your web server, the configuration would need to contain (source):

<system.webServer>
  <rewrite>
    <outboundRules>
      <rule name="Set custom HTTP response header">
        <match serverVariable="RESPONSE_X_Robots_Tag" pattern=".*" />
        <conditions>
          <add input="{REQUEST_URI}" pattern="^/txtfiledir/.*\.txt$" />
        </conditions>
        <action type="Rewrite" value="noindex"/>
      </rule>
    </outboundRules>
  </rewrite>
</system.webServer>
| improve this answer | |
  • Is there something I'm missing in the question that implies the OP is using an Apache web server? – David Z Oct 27 at 0:08
  • @DavidZ Well.. 'turns out it was a good guess : ) – Bangkokian Oct 27 at 9:32
  • 1
    Apache is the most common web server, especially for beginner level shared hosting, so it is always a good guess. I have added nginx and IIS configuration as well so that question will be useful to more people in the future. – Stephen Ostermiller Oct 27 at 9:36
  • Sure, and thanks for the edit. I just brought it up out of concern for people who don't use Apache and might be confused when they are told to look for a .htaccess file and don't find it; or more generally, whenever instructions for the most common version of a thing are presented as though they apply to all versions of the thing. – David Z Oct 27 at 21:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.