How to block user agent from accessing my WordPress site with htaccess?

Question

I have noticed large amount of traffic coming to my site from different IP's with user agent.

Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)

I don't have much experience with Apache servers and hope someone can explain to me how to actually block this from accessing my site with .htaccess file.

I tried with

SetEnvIfNoCase User-Agent "^siteexplorer?$" bad_user
Deny from env=bad_user

but it's not working.

Answer 1

Your regular expression specifies that the string must not have anything before or after the name of the crawler. That is what the ^ and $ do. I'm also not sure why you would have a ? in there, which makes the "r" at the end optional.

Try this instead:

SetEnvIfNoCase User-Agent "siteexplorer" bad_user
Deny from env=bad_user

Answer 2

It is more than okay to block any agent when the agent is very specific. In this case, you should be safe.

Here is an example:

RewriteCond %{HTTP_USER_AGENT} ^.*SiteExplorer.*$ [NC]
RewriteRule .* - [F,L]

You will notice that I added .* to wildcard any number of characters.

I use this method on a long list of agents where (agent2-regex|agent2-regex|agent3-regex) is used to make multiple comparisons. The () brackets the conditions and the | is an OR operator.