Tagged Questions

0
votes
1answer
242 views

Apache real time robot detection

Background Our website is being a. scraped - complete pages downloaded b. spidered - complete pages and sub links followed c. mined - specific files downloaded at high speed by hundreds of ...
8
votes
3answers
287 views

Tactics for dealing with misbehaving robots

I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site. Obviously we've had ...