7
The solution is the same as for X-Robots noindex a folder of PDFs and Prevent XML sitemaps from showing up in Google search results. You use the X-Robots-Tag HTTP header rather than a meta tag. The HTTP header served with the txt file should look like:
X-Robots-Tag: noindex
After implementing the HTTP header, remove the disallow from your robots.txt. ...
2
Is this a proper method for blocking bad bots?
Well, it is one method. (Although you wouldn't block whole user-agent strings using regex alternation; if that is what you are implying by your CondPattern?)
However, "bad bots" often do not disclose themselves using an easily identifiable user-agent. They may use a standard browser user agent, which ...
Only top voted, non community-wiki answers of a minimum length are eligible
