How do I "noindex" a text (.txt) file?

Question

I have a directory full of .txt files that are used as part of a larger javascript application. The text files are not inteded to be directly accessed, and are used as includes on separate pages as part of the application functionality).

So.. I added the directory to my robots.txt file because I don't want the files to show up independently in search results.

However, robots.txt simply makes the .txt files inaccessible to robots. It doesn't actually make the .txt files un-indexable.

What I really want to do is "noindex" the .txt files. But .txt files aren't like .html files.. a noindex meta tag will be visible in the text.

So how does one "noindex" a file that's raw text without corrupting the text with a tag?

Is there some other way to noindex a text file, externally? Or noindex the overall directory?

Answer 1

The solution is the same as for X-Robots noindex a folder of PDFs and Prevent XML sitemaps from showing up in Google search results. You use the X-Robots-Tag HTTP header rather than a meta tag. The HTTP header served with the txt file should look like:

X-Robots-Tag: noindex

After implementing the HTTP header, remove the disallow from your robots.txt. In order for crawlers to be able to see and honor the header, you need to make the txt files crawlable.

---

Under Apache web servers (most shared hosting), put the following code in the .htaccess file in the directory containing the .txt files:

<FilesMatch ".txt$">
  Header set X-Robots-Tag "noindex"
</FilesMatch>

---

If you are using nginx, your configuration would need to contain (source):

location ~* ^/txtfiledir/.*\.txt$ {
  add_header  X-Robots-Tag "noindex";
}

---

If you are using IIS as your web server, the configuration would need to contain (source):

<system.webServer>
  <rewrite>
    <outboundRules>
      <rule name="Set custom HTTP response header">
        <match serverVariable="RESPONSE_X_Robots_Tag" pattern=".*" />
        <conditions>
          <add input="{REQUEST_URI}" pattern="^/txtfiledir/.*\.txt$" />
        </conditions>
        <action type="Rewrite" value="noindex"/>
      </rule>
    </outboundRules>
  </rewrite>
</system.webServer>