So.. I added the directory to my robots.txt file because I don't want the files to show up independently in search results.
However, robots.txt simply makes the .txt files inaccessible to robots. It doesn't actually make the .txt files un-indexable.
What I really want to do is "noindex" the .txt files. But .txt files aren't like .html files.. a noindex meta tag will be visible in the text.
So how does one "noindex" a file that's raw text without corrupting the text with a tag?
Is there some other way to noindex a text file, externally? Or noindex the overall directory?