7

The solution is the same as for X-Robots noindex a folder of PDFs and Prevent XML sitemaps from showing up in Google search results. You use the X-Robots-Tag HTTP header rather than a meta tag. The HTTP header served with the txt file should look like: X-Robots-Tag: noindex After implementing the HTTP header, remove the disallow from your robots.txt. ...


2

Is this a proper method for blocking bad bots? Well, it is one method. (Although you wouldn't block whole user-agent strings using regex alternation; if that is what you are implying by your CondPattern?) However, "bad bots" often do not disclose themselves using an easily identifiable user-agent. They may use a standard browser user agent, which ...


Only top voted, non community-wiki answers of a minimum length are eligible