I have apache logs from my server, and I filter out visits from bots/crawlers/scrapers using a python script that checks for user agent strings containing text like 'bot', 'googlebot', etc.
Lately, there have been a number of visits to my site from what I believe is msnbot/bingbot, but they don't report it in their user agent.
An example of a log line is:
18.104.22.168 - - [27/May/2011:07:45:07 -0400] ...stuff... "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/5.0; SLCC1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648)"
And a reverse DNS of the IP address:
Name: msnbot-207-46-12-74.search.msn.com Address: 22.214.171.124
Right now I'm thinking of filtering out specific IP addresses that I detect are
msnbot when I perform rDNS on their IP addresses, but when I presented this to a coworker, he felt that maybe there was something else to the fact that Microsoft's bingbots weren't reporting their user agent, like if it was IE private browsing or bing's safe-website crawler.
I've looked on Project Honeypot, various sites that have databases of user agents, and have confirmed net blocks of IP addresses that MSN/Bing bots use, but I think he wants even stricter confirmation.
Anyone know the behavior of these bots and why they're not reporting their agent strings as being 'bots'?