Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have noticed large amount of traffic coming to my site from different IP's with user agent.

Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)

I don't have much experience with Apache servers and hope someone can explain to me how to actually block this from accessing my site with .htaccess file.

I tried with

SetEnvIfNoCase User-Agent "^siteexplorer?$" bad_user
Deny from env=bad_user

but it's not working.

share|improve this question
Blocking a site access by user-agent is a bad idea because you block all users with this user-agent (especially if the user-agent is well known like Mozilla). Try to block IP adresses instead. – Zistoloen Mar 14 '14 at 11:51
it's robot that changes ip every few hours, so ip based block won't work for me... i don't wanna block mozilla, just siteexplorer... – onedevteam.com Mar 14 '14 at 12:23

Your regular expression specifies that the string must not have anything before or after the name of the crawler. That is what the ^ and $ do. I'm also not sure why you would have a ? in there, which makes the "r" at the end optional.

Try this instead:

SetEnvIfNoCase User-Agent "siteexplorer" bad_user
Deny from env=bad_user
share|improve this answer

It is more than okay to block any agent when the agent is very specific. In this case, you should be safe.

Here is an example:

RewriteCond %{HTTP_USER_AGENT} ^.*SiteExplorer.*$ [NC]
RewriteRule .* - [F,L]

You will notice that I added .* to wildcard any number of characters.

I use this method on a long list of agents where (agent2-regex|agent2-regex|agent3-regex) is used to make multiple comparisons. The () brackets the conditions and the | is an OR operator.

share|improve this answer
Note that ^.*SiteExplorer.*$ is the same as simply specifying SiteExplorer (ie. match "SiteExplorer" anywhere inside the user agent string). – w3dk Mar 14 '14 at 23:31
True. I thought I would correct his usage as it appeared. I prefer the shorter method you pointed out, but I have also made mistakes this way too. – closetnoc Mar 14 '14 at 23:54

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.