How to detect search engine bots reliably within asp.net website?

Question

What is the best way to detect bot generated activity (clicks/page visits) etc on a asp.net website? We have a website where we track leads generated to external websites, we track the IP of the user that generates the lead however we are seeing a lot of leads generated by Google and other search bots. What is the best way to filter this activity. I have heard of testing user agent strings and filtering based on know IP addresses, both available as data dumps from various sources, I am not sure which is the best one to use.

Thank you.

Answer 1

Like @Kinopiko said, Bots, esepcialy the top ones like google, or bing leave a clear UserAgent.

I don't know what your code looks like so I can't tell you what to do but to find the UserAgent in Asp.Net you look at Request.UserAgent in any WebForm, CodeBehind, or MVC Controller.

Answer 2

Sorry but I have no idea about asp.net websites, but any reputable search engine will be sending you a user agent string which tells you it's a bot, and that should be available in your log file. Another giveaway is that they look for /robots.txt.

My checker looks like this (Perl):

sub is_bot
{
    my ($user_agent) = @_;
    if ($user_agent =~
    /msnbot
        |www\.cuil\.com
    |Yahoo!\s+Slurp
    |Googlebot
    |Speedy\sSpider
    |MLBot
    |princeton crawler
    |accelobot
    |crawler\@dotnetdotcom
    |help\.naver\.com
    |GingerCrawler
    |Sosospider
    |www.exabot.com
    |Baiduspider
    |Ask\sJeeves
    |Java\/
    |telehouse\.ru
    |Tagoobot
    |Baypup
    |SimilarPages
    |Spinn3r
    |VoilaBot
    |Yandex
    |Xenu\sLink\sSleuth
    |www\.searchme\.com
    |MJ12bot
    |kilomonkey\.com
    |Mediapartners-Google
    |Sogou\sweb\sspider
    |YoudaoBot
    |seexie\.com
    |Yahoo.*Slurp
    |YahooCacheSystem
    |crawler\@nutch\.biz
    |psbot
    |ia_archiver-web\.archive\.org
    |sbider
    |xrss\.eu
    |scoutjet
        |www\.puritysearch\.net
        |Bing
        |BaiduImagespider
        |baidu\.jp
        |facebookexternalhit
        |ssllabs\.com
        |Python-urllib
        |drupal\.org
        |HTTrack
        |Willow\s+Internet\s+Crawler\s+by\s+Twotrees
    /x) {
    return 1;
    }
    return;
}

If you look in the source code of awstats you'll find something better than the above.