Apache real time robot detection

Question

Background

Our website is being

a. scraped - complete pages downloaded
b. spidered - complete pages and sub links followed
c. mined - specific files downloaded at high speed

by hundreds of robots - some off the shelf and some home made

We have another site especially for the robots to go to which offers them easier to parse data (XML/JSON) but they are stubbornly ignoring that site and our fair-use charter so all bets are off.

What we cannot do

It is not possible to ask the users to signup/signin - the data is public and will remain so, we just do not want the robots to bother the real users and hammer the site.

We cannot use reCaptcha or any other external service.

Middletier is JAVA and I would prefer something on the Apache frontend

What have we done

1. robots.txt has

   User-agent: *
Disallow: /

2. We have captchas in place to stop complete file download but the robots just take the non-capchaed page-by-page and likely stitch them together at their leisure.

A normal and acceptable user will come in, do a search, load the resulting document, perhaps download a PDF - all surrounded by image files, CSS files and script requests - some/most giving 304s after the first visit. If these users are clicking very fast through the result list, they may get the "are you a robot" captcha.

The mining robot will typically only request one type of URL at high speed if too fast, we also give him a captcha that he may or may not OCR or send to a human user to be deciphered.

Question 1: Do you know if jCaptcha has been compromised and is there a better JAVA based alternative (simple captcha for example)

Question 2: Is there a good way to detect and bar a robot at the Apache end by examining the URL pattern?

I would like to block at least the type c robot. It comes in with the complete URL of the page in question and does not load any auxiliary files like CSS, images or scripts.

I was thinking about monitoring all the requests per IP (but will that fail with IPv6?), if 10 of the same type of requests without any other (regex?) block the robot - but that raises the question on how to not block a user with a well cached browser - for example insert something that is not cached by a normal browser (idea from Bill Rowe)

I do not currently have the skills myself to write an Apache module so I was hoping some open source that I could not find when I Googled is known by some of you.

Answer 1

This may seem a little ridiculous, but if you're getting the traffic you say you are from the SAME IP addresses, would it be possible to install Fail2Ban on the server itself? (not Apache module). Some info: http://www.fail2ban.org/wiki/index.php/Main_Page but this scans Error logs from the server (Apache's logs, for example) and finds patterns where someone has a lot of 40* errors, and so forth.

As for a .htaccess solution, (from http://www.askapache.com/htaccess/blocking-bad-bots-and-scrapers-with-htaccess.html)

ErrorDocument 403 /403.html

RewriteEngine On
RewriteBase /

# IF THE UA STARTS WITH THESE
RewriteCond %{HTTP_USER_AGENT} ^(aesop_com_spiderman|alexibot|backweb|bandit|batchftp|bigfoot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(black.?hole|blackwidow|blowfish|botalot|buddy|builtbottough|bullseye) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(cheesebot|cherrypicker|chinaclaw|collector|copier|copyrightcheck) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(cosmos|crescent|curl|custo|da|diibot|disco|dittospyder|dragonfly) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(drip|easydl|ebingbong|ecatch|eirgrabber|emailcollector|emailsiphon) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(emailwolf|erocrawler|exabot|eyenetie|filehound|flashget|flunky) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(frontpage|getright|getweb|go.?zilla|go-ahead-got-it|gotit|grabnet) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(grafula|harvest|hloader|hmview|httplib|httrack|humanlinks|ilsebot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(infonavirobot|infotekies|intelliseek|interget|iria|jennybot|jetcar) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(joc|justview|jyxobot|kenjin|keyword|larbin|leechftp|lexibot|lftp|libweb) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(likse|linkscan|linkwalker|lnspiderguy|lwp|magnet|mag-net|markwatch) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(mata.?hari|memo|microsoft.?url|midown.?tool|miixpc|mirror|missigua) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(mister.?pix|moget|mozilla.?newt|nameprotect|navroad|backdoorbot|nearsite) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(net.?vampire|netants|netcraft|netmechanic|netspider|nextgensearchbot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(attach|nicerspro|nimblecrawler|npbot|octopus|offline.?explorer) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(offline.?navigator|openfind|outfoxbot|pagegrabber|papa|pavuk) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(pcbrowser|php.?version.?tracker|pockey|propowerbot|prowebwalker) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(psbot|pump|queryn|recorder|realdownload|reaper|reget|true_robot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(repomonkey|rma|internetseer|sitesnagger|siphon|slysearch|smartdownload) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(snake|snapbot|snoopy|sogou|spacebison|spankbot|spanner|sqworm|superbot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(superhttp|surfbot|asterias|suzuran|szukacz|takeout|teleport) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(telesoft|the.?intraformant|thenomad|tighttwatbot|titan|urldispatcher) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(turingos|turnitinbot|urly.?warning|vacuum|vci|voideye|whacker) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(libwww-perl|widow|wisenutbot|wwwoffle|xaldon|xenu|zeus|zyborg|anonymouse) [NC,OR]

# STARTS WITH WEB
RewriteCond %{HTTP_USER_AGENT} ^web(zip|emaile|enhancer|fetch|go.?is|auto|bandit|clip|copier|master|reaper|sauger|site.?quester|whack) [NC,OR]

# ANYWHERE IN UA -- GREEDY REGEX
RewriteCond %{HTTP_USER_AGENT} ^.*(craftbot|download|extract|stripper|sucker|ninja|clshttp|webspider|leacher|collector|grabber|webpictures).*$ [NC]

# ISSUE 403 / SERVE ERRORDOCUMENT
RewriteRule . - [F,L]

This is definitely not comprehensive, so it may require some tweaking.