Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have been working more on the bot filter for my website, but by no means is it complete.

So far I have the main ones:

Google, Yahoo, MSN, Baidu, Amazon, and a few others...

Right now, I am using a filter to compare the referral URL, existing domain, and known browsers vs. non-browser useragents.

Are there any other good techniques to detect if the hit is coming from a bot or not?

share|improve this question
add comment

1 Answer

up vote 1 down vote accepted

Are there any other good techniques to detect if the hit is coming from a bot or not?

It depends upon which variety of bot you're hunting - here are a few tips for isolating malicious bots:

  • Look for hits on non-existent (or restricted access) administrative scripts, e-mail scripts, et cetera
  • Look for nearly-instantaneous retrieval of your site content
  • Look for repeated hits on your feeds (particularly if you know that your content is being used on scraper sites)

You might also check out user-agent.org for some of the more obscure search bot agents you can expect to see and review the Where can I find a list of search engine crawler user agents and their domain names? thread started by Dev a few days ago.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.