3

I have been working more on the bot filter for my website, but by no means is it complete.

So far I have the main ones:

Google, Yahoo, MSN, Baidu, Amazon, and a few others...

Right now, I am using a filter to compare the referral URL, existing domain, and known browsers vs. non-browser useragents.

Are there any other good techniques to detect if the hit is coming from a bot or not?

1

Are there any other good techniques to detect if the hit is coming from a bot or not?

It depends upon which variety of bot you're hunting - here are a few tips for isolating malicious bots:

  • Look for hits on non-existent (or restricted access) administrative scripts, e-mail scripts, et cetera
  • Look for nearly-instantaneous retrieval of your site content
  • Look for repeated hits on your feeds (particularly if you know that your content is being used on scraper sites)

You might also check out user-agent.org for some of the more obscure search bot agents you can expect to see and review the Where can I find a list of search engine crawler user agents and their domain names? thread started by Dev a few days ago.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.