Our website is being
a. scraped - complete pages downloaded
b. spidered - complete pages and sub links followed
c. mined - specific files downloaded at high speed
by hundreds of robots - some off the shelf and some home made
We have another site especially for the robots to go to which offers them easier to parse data (XML/JSON) but they are stubbornly ignoring that site and our fair-use charter so all bets are off.
What we cannot do
It is not possible to ask the users to signup/signin - the data is public and will remain so, we just do not want the robots to bother the real users and hammer the site.
We cannot use reCaptcha or any other external service.
Middletier is JAVA and I would prefer something on the Apache frontend
What have we done
captchasin place to stop complete file download but the robots just take the non-capchaed page-by-page and likely stitch them together at their leisure.
A normal and acceptable user will come in, do a search, load the resulting document, perhaps download a PDF - all surrounded by image files, CSS files and script requests - some/most giving 304s after the first visit. If these users are clicking very fast through the result list, they may get the "are you a robot" captcha.
The mining robot will typically only request one type of URL at high speed if too fast, we also give him a captcha that he may or may not OCR or send to a human user to be deciphered.
Question 1: Do you know if jCaptcha has been compromised and is there a better JAVA based alternative (simple captcha for example)
Question 2: Is there a good way to detect and bar a robot at the Apache end by examining the URL pattern?
I would like to block at least the type c robot. It comes in with the complete URL of the page in question and does not load any auxiliary files like CSS, images or scripts.
I was thinking about monitoring all the requests per IP (but will that fail with IPv6?), if 10 of the same type of requests without any other (regex?) block the robot - but that raises the question on how to not block a user with a well cached browser - for example insert something that is not cached by a normal browser (idea from Bill Rowe)
I do not currently have the skills myself to write an Apache module so I was hoping some open source that I could not find when I Googled is known by some of you.