1

I am proposing blocking bad bots using the following code in /etc/sites-enabled/site.conf and I have a few questions.

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} useragent1|useragent2|useragent3 [NC]
RewriteRule .* - [F,L]
  1. Is this a proper method for blocking bad bots?
  2. Will this have any impact on legitimate users who are withholding their user-agent?
  3. Should I be using .* to finalise the rule or ^ or .^?

Any other suggestions or comments very much appreciated and thank you for your time.

2
  1. Is this a proper method for blocking bad bots?

Well, it is one method. (Although you wouldn't block whole user-agent strings using regex alternation; if that is what you are implying by your CondPattern?)

However, "bad bots" often do not disclose themselves using an easily identifiable user-agent. They may use a standard browser user agent, which will obviously render this method useless.

There might also be too many "bad bots" to make this method scalable. It's potentially difficult to maintain unless you are OK with just blocking a few known "baddies".

  1. Will this have any impact on legitimate users who are withholding their user-agent?

"Legitimate users" shouldn't "withhold their user-agent". It is generally OK to block visitors with an empty user-agent (if that's what you mean by "withholding"). If a "legitimate user" changes their user-agent to mimic a "bad bot" then they can expect to be blocked.

As mentioned above, you can't block "bad bots" that are pretending to be real users (ie. using a standard browser user-agent string), as you will obviously block real users as well.

  1. Should I be using .* to finalise the rule or ^ or .^?

From a regex point of view, ^ is more efficient in this instance, since you don't need to actually match anything, you just need to be successful for every request.

.* unnecessary traverses the entire URL-path and discards the result. Whereas ^ simply asserts the start-of-string and matches nothing.

.^ doesn't make sense.

[F,L] - You don't need the L here - it is implied when using the F flag.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.