2

I got a hit from a crawler with a user-agent called DV CRAWLER which is an abvious a spam-bot. I tried to block it in both .htacess and nginx configuration as I'm running nginx as a reverse proxy in front of apache.

Here is the code I used for .htaccess

RewriteCond %{HTTP_USER_AGENT} ^.*(Baiduspider|DV CRAWLER).*$ [NC]
RewriteRule .* - [F,L]

Seems that the spacing in the name of the user agent has broke the code. I discovered that it only works with user agents that has no spaces. Same scenario with nginx, it doesn't accept spacing in the name of user agent and returns error.

Nginx code:

if ($http_user_agent ~ (Baiduspider|DV CRAWLER) ) {
return 403;
}

So, what is the alternative for this? I don't want these spam bots to crawl my website. Any answer would be greatly appreciated.

  • I can't say exactly why that's failing, but you could try: (Baiduspider|DV[ |\s|%20]CRAWLER) for a few alternatives. Or log the header and make sure that's exactly what it is (i.e. it's not being encoded) – jhenn May 29 '15 at 17:58
  • @jhenn %20 in this context would try to match the literal string "%20". – MrWhite Jun 1 '15 at 7:37
1

Your regex code in general is wrong.

Try instead something like this:

RewriteCond %{HTTP_USER_AGENT} (.*Baiduspider.*|.*DV.*CRAWLER.*) [NC]

You are matching against a string in each iteration between the parenthesis () separated by the pipe character | whereas .* is a wild card that matches anything. Optionally you can use \s or \s+ for spaces but .* works too and may be better. Not knowing what the DV CRAWLER string looks like, I made a guess (SWAG). You may need to adjust this.

For example: A string of a line of red cars driving down the street could be matched simply using .*red.*cars.*. There are slicker regular expressions for this, but using this simple method can be safely repeated over and over.

  • Thank you, that helped me to understand. Can I use the .* and \s in nginx too? – Mina Hafzalla May 29 '15 at 18:41
  • @JerryB Yes. Both use regular expressions. Here is the wikipedia page on regex: en.wikipedia.org/wiki/Regular_expression It takes a bit to get into, but you will get it okay. It can make you go cross-eyed, but stick with it! – closetnoc May 29 '15 at 21:04
3

The space is a delimiter (ie. a special character) in .htaccess so must be backslash escaped if you want to match a literal space in the regex. Eg. DV\ CRAWLER. (Otherwise you are likely to get a less than helpful 500 Internal Server error.)

Or, you can use the shorthand character class \s which matches any white space character (space, tab or new line / line break) - so not technically just a space.

  • 2
    This is the answer I would accept if it were my question. I couldn't figure out why the space would have special meaning in regex, but you found the reason: its embedded in a file in which the space has special meaning. – Stephen Ostermiller Jun 1 '15 at 9:50
1

When in doubt, add parenthesis and escaping to regular expressions. Try this first:

(Baiduspider|(DV CRAWLER))

I think that your problem is that it evaluating as "Baiduspider or DV followed by CRAWLER" when you don't have the paranthesis. If that doesn't work, then try escaping the space:

(Baiduspider|(DV\sCRAWLER))

Where \s is any white space character.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.