Possible Duplicate:
How To Track Down and Stop Rogue Bots?

I had an incident yesterday where my web site was taken down by a web crawler that was ignoring my robots.txt. I'm pretty sure nothing malicious was intended, but the crawler wandered into pages that completely overloaded my database with time consuming requests.

What to do for "next time"?

(1) Obviously make it harder to trigger onerous requests by just clicking on a link.

(2) The other idea I have is to add "poison pill" links that would be invisible to humans, but would have a side effect of marking the IP address as a robot.

The question is, would this be likely to trigger false positives, for example because the browser was trying to prefetch the "mark-me-as-a-robot" link.

  • 1
    See this question for some options: stackoverflow.com/q/233192/902654 (I realize that the intent of the crawler is different, but the same techniques can be used)
    – someone
    Sep 27 '11 at 23:20

Well... One thing I did once was use perl to set up a link to an endlessly iterating link, making sure that a new portion of the URL was generated each time, like this:


There are other alternatives along those lines, but you get the idea.

  • 3
    wouldn't this turn a web crawler into a flat out denial of service attack? The idea is to make the robot go away, not to make it stick around forever.
    – ddyer
    Sep 27 '11 at 23:30
  • It was a pretty crappy crawler. They only ran it on the site twice after I did this, and then they went away. I figure they only hit up our small site for a few databases that we had.
    – Albert Perrien
    Sep 27 '11 at 23:45

Not the answer you're looking for? Browse other questions tagged or ask your own question.