Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Possible Duplicate:
How To Track Down and Stop Rogue Bots?

I had an incident yesterday where my web site was taken down by a web crawler that was ignoring my robots.txt. I'm pretty sure nothing malicious was intended, but the crawler wandered into pages that completely overloaded my database with time consuming requests.

What to do for "next time"?

(1) Obviously make it harder to trigger onerous requests by just clicking on a link.

(2) The other idea I have is to add "poison pill" links that would be invisible to humans, but would have a side effect of marking the IP address as a robot.

The question is, would this be likely to trigger false positives, for example because the browser was trying to prefetch the "mark-me-as-a-robot" link.

share|improve this question

migrated from stackoverflow.com Sep 29 '11 at 11:27

This question came from our site for professional and enthusiast programmers.

marked as duplicate by John Conde Sep 29 '11 at 11:42

This question was marked as an exact duplicate of an existing question.

See this question for some options: stackoverflow.com/q/233192/902654 (I realize that the intent of the crawler is different, but the same techniques can be used) – someone Sep 27 '11 at 23:20

Well... One thing I did once was use perl to set up a link to an endlessly iterating link, making sure that a new portion of the URL was generated each time, like this:


There are other alternatives along those lines, but you get the idea.

share|improve this answer
wouldn't this turn a web crawler into a flat out denial of service attack? The idea is to make the robot go away, not to make it stick around forever. – ddyer Sep 27 '11 at 23:30
It was a pretty crappy crawler. They only ran it on the site twice after I did this, and then they went away. I figure they only hit up our small site for a few databases that we had. – Albert Perrien Sep 27 '11 at 23:45

Not the answer you're looking for? Browse other questions tagged or ask your own question.