I have noticed large amount of traffic coming to my site from different IP's with user agent.

Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)

I don't have much experience with Apache servers and hope someone can explain to me how to actually block this from accessing my site with .htaccess file.

I tried with

SetEnvIfNoCase User-Agent "^siteexplorer?$" bad_user
Deny from env=bad_user

but it's not working.

  • Blocking a site access by user-agent is a bad idea because you block all users with this user-agent (especially if the user-agent is well known like Mozilla). Try to block IP adresses instead.
    – Zistoloen
    Mar 14, 2014 at 11:51
  • 2
    it's robot that changes ip every few hours, so ip based block won't work for me... i don't wanna block mozilla, just siteexplorer... Mar 14, 2014 at 12:23

2 Answers 2


Your regular expression specifies that the string must not have anything before or after the name of the crawler. That is what the ^ and $ do. I'm also not sure why you would have a ? in there, which makes the "r" at the end optional.

Try this instead:

SetEnvIfNoCase User-Agent "siteexplorer" bad_user
Deny from env=bad_user

It is more than okay to block any agent when the agent is very specific. In this case, you should be safe.

Here is an example:

RewriteCond %{HTTP_USER_AGENT} ^.*SiteExplorer.*$ [NC]
RewriteRule .* - [F,L]

You will notice that I added .* to wildcard any number of characters.

I use this method on a long list of agents where (agent2-regex|agent2-regex|agent3-regex) is used to make multiple comparisons. The () brackets the conditions and the | is an OR operator.

  • Note that ^.*SiteExplorer.*$ is the same as simply specifying SiteExplorer (ie. match "SiteExplorer" anywhere inside the user agent string).
    – MrWhite
    Mar 14, 2014 at 23:31
  • True. I thought I would correct his usage as it appeared. I prefer the shorter method you pointed out, but I have also made mistakes this way too.
    – closetnoc
    Mar 14, 2014 at 23:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.