Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Background

Our website is being

a. scraped - complete pages downloaded
b. spidered - complete pages and sub links followed
c. mined - specific files downloaded at high speed

by hundreds of robots - some off the shelf and some home made

We have another site especially for the robots to go to which offers them easier to parse data (XML/JSON) but they are stubbornly ignoring that site and our fair-use charter so all bets are off.

What we cannot do

It is not possible to ask the users to signup/signin - the data is public and will remain so, we just do not want the robots to bother the real users and hammer the site.

We cannot use reCaptcha or any other external service.

Middletier is JAVA and I would prefer something on the Apache frontend

What have we done

  1. robots.txt has

    User-agent: *
    Disallow: /

  2. We have captchas in place to stop complete file download but the robots just take the non-capchaed page-by-page and likely stitch them together at their leisure.

A normal and acceptable user will come in, do a search, load the resulting document, perhaps download a PDF - all surrounded by image files, CSS files and script requests - some/most giving 304s after the first visit. If these users are clicking very fast through the result list, they may get the "are you a robot" captcha.

The mining robot will typically only request one type of URL at high speed if too fast, we also give him a captcha that he may or may not OCR or send to a human user to be deciphered.

Question 1: Do you know if jCaptcha has been compromised and is there a better JAVA based alternative (simple captcha for example)

Question 2: Is there a good way to detect and bar a robot at the Apache end by examining the URL pattern?

I would like to block at least the type c robot. It comes in with the complete URL of the page in question and does not load any auxiliary files like CSS, images or scripts.

I was thinking about monitoring all the requests per IP (but will that fail with IPv6?), if 10 of the same type of requests without any other (regex?) block the robot - but that raises the question on how to not block a user with a well cached browser - for example insert something that is not cached by a normal browser (idea from Bill Rowe)

I do not currently have the skills myself to write an Apache module so I was hoping some open source that I could not find when I Googled is known by some of you.

share|improve this question

1 Answer 1

This may seem a little ridiculous, but if you're getting the traffic you say you are from the SAME IP addresses, would it be possible to install Fail2Ban on the server itself? (not Apache module). Some info: http://www.fail2ban.org/wiki/index.php/Main_Page but this scans Error logs from the server (Apache's logs, for example) and finds patterns where someone has a lot of 40* errors, and so forth.

As for a .htaccess solution, (from http://www.askapache.com/htaccess/blocking-bad-bots-and-scrapers-with-htaccess.html)

ErrorDocument 403 /403.html

RewriteEngine On
RewriteBase /

# IF THE UA STARTS WITH THESE
RewriteCond %{HTTP_USER_AGENT} ^(aesop_com_spiderman|alexibot|backweb|bandit|batchftp|bigfoot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(black.?hole|blackwidow|blowfish|botalot|buddy|builtbottough|bullseye) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(cheesebot|cherrypicker|chinaclaw|collector|copier|copyrightcheck) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(cosmos|crescent|curl|custo|da|diibot|disco|dittospyder|dragonfly) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(drip|easydl|ebingbong|ecatch|eirgrabber|emailcollector|emailsiphon) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(emailwolf|erocrawler|exabot|eyenetie|filehound|flashget|flunky) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(frontpage|getright|getweb|go.?zilla|go-ahead-got-it|gotit|grabnet) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(grafula|harvest|hloader|hmview|httplib|httrack|humanlinks|ilsebot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(infonavirobot|infotekies|intelliseek|interget|iria|jennybot|jetcar) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(joc|justview|jyxobot|kenjin|keyword|larbin|leechftp|lexibot|lftp|libweb) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(likse|linkscan|linkwalker|lnspiderguy|lwp|magnet|mag-net|markwatch) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(mata.?hari|memo|microsoft.?url|midown.?tool|miixpc|mirror|missigua) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(mister.?pix|moget|mozilla.?newt|nameprotect|navroad|backdoorbot|nearsite) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(net.?vampire|netants|netcraft|netmechanic|netspider|nextgensearchbot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(attach|nicerspro|nimblecrawler|npbot|octopus|offline.?explorer) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(offline.?navigator|openfind|outfoxbot|pagegrabber|papa|pavuk) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(pcbrowser|php.?version.?tracker|pockey|propowerbot|prowebwalker) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(psbot|pump|queryn|recorder|realdownload|reaper|reget|true_robot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(repomonkey|rma|internetseer|sitesnagger|siphon|slysearch|smartdownload) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(snake|snapbot|snoopy|sogou|spacebison|spankbot|spanner|sqworm|superbot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(superhttp|surfbot|asterias|suzuran|szukacz|takeout|teleport) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(telesoft|the.?intraformant|thenomad|tighttwatbot|titan|urldispatcher) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(turingos|turnitinbot|urly.?warning|vacuum|vci|voideye|whacker) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(libwww-perl|widow|wisenutbot|wwwoffle|xaldon|xenu|zeus|zyborg|anonymouse) [NC,OR]

# STARTS WITH WEB
RewriteCond %{HTTP_USER_AGENT} ^web(zip|emaile|enhancer|fetch|go.?is|auto|bandit|clip|copier|master|reaper|sauger|site.?quester|whack) [NC,OR]

# ANYWHERE IN UA -- GREEDY REGEX
RewriteCond %{HTTP_USER_AGENT} ^.*(craftbot|download|extract|stripper|sucker|ninja|clshttp|webspider|leacher|collector|grabber|webpictures).*$ [NC]

# ISSUE 403 / SERVE ERRORDOCUMENT
RewriteRule . - [F,L]

This is definitely not comprehensive, so it may require some tweaking.

share|improve this answer
2  
If homemade bots are an issue the user-agent can't be trusted ... but, if you must go that route, don't forget wget. –  danlefree Jun 13 '12 at 16:58
    
Use an Apache log analyzer if there are custom user agents. You could even set up a PHP script that tracks all unique user agents and saves them to either a flat file or a database, and then sends a 403 error to any that are not of "the common browsers." Thanks for the wget suggestion though. –  ionFish Jun 13 '12 at 18:31
    
Our log files are 1GB zipped per day. –  mplungjan Jun 13 '12 at 21:39
    
Thanks for the answer, I will look into what you suggest. The IPs are the same for a period, then they change. Around 30K unique IPs a day. I wonder if the htaccess thing would slow things down too much unless of course we can use it to permanently ban them –  mplungjan Jun 13 '12 at 21:40
    
I say try it, one one of our servers we have approximately 140 domains, most of which have complex .htaccess rules for the applications they run. You could even put those rules into the Apache config itself, so that it's loaded into memory when Apache is started and not read each time a site is accessed. –  ionFish Jun 14 '12 at 0:43

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.