Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

What is the best way to detect bot generated activity (clicks/page visits) etc on a asp.net website? We have a website where we track leads generated to external websites, we track the IP of the user that generates the lead however we are seeing a lot of leads generated by Google and other search bots. What is the best way to filter this activity. I have heard of testing user agent strings and filtering based on know IP addresses, both available as data dumps from various sources, I am not sure which is the best one to use.

Thank you.

share|improve this question
add comment

2 Answers 2

Like @Kinopiko said, Bots, esepcialy the top ones like google, or bing leave a clear UserAgent.

I don't know what your code looks like so I can't tell you what to do but to find the UserAgent in Asp.Net you look at Request.UserAgent in any WebForm, CodeBehind, or MVC Controller.

share|improve this answer
    
Thank you, my code in asp.net looks very similar to the check shown below. The reason I was doubtful about the reliability of my check is that some sites are selling lists of known bot user agents that contains more than a few thousand bots, however based on replies here I feel checking for the top bots should suffice. –  user1081 Jul 28 '10 at 8:00
add comment

Sorry but I have no idea about asp.net websites, but any reputable search engine will be sending you a user agent string which tells you it's a bot, and that should be available in your log file. Another giveaway is that they look for /robots.txt.

My checker looks like this (Perl):

sub is_bot
{
    my ($user_agent) = @_;
    if ($user_agent =~
    /msnbot
        |www\.cuil\.com
    |Yahoo!\s+Slurp
    |Googlebot
    |Speedy\sSpider
    |MLBot
    |princeton crawler
    |accelobot
    |crawler\@dotnetdotcom
    |help\.naver\.com
    |GingerCrawler
    |Sosospider
    |www.exabot.com
    |Baiduspider
    |Ask\sJeeves
    |Java\/
    |telehouse\.ru
    |Tagoobot
    |Baypup
    |SimilarPages
    |Spinn3r
    |VoilaBot
    |Yandex
    |Xenu\sLink\sSleuth
    |www\.searchme\.com
    |MJ12bot
    |kilomonkey\.com
    |Mediapartners-Google
    |Sogou\sweb\sspider
    |YoudaoBot
    |seexie\.com
    |Yahoo.*Slurp
    |YahooCacheSystem
    |crawler\@nutch\.biz
    |psbot
    |ia_archiver-web\.archive\.org
    |sbider
    |xrss\.eu
    |scoutjet
        |www\.puritysearch\.net
        |Bing
        |BaiduImagespider
        |baidu\.jp
        |facebookexternalhit
        |ssllabs\.com
        |Python-urllib
        |drupal\.org
        |HTTrack
        |Willow\s+Internet\s+Crawler\s+by\s+Twotrees
    /x) {
    return 1;
    }
    return;
}

If you look in the source code of awstats you'll find something better than the above.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.