I've been experiencing high RPM spikes recently. Something like this: When I debugged, I've found reasons to believe the reason is the msnbot suddenly makes a massive crawl and then stops. I assume ...
A .NET site I am currently working on is being hit rather frequently by the Bingbot on the path: /:80/robots.txt This is obviously throwing the following exception: System.Web.HttpException: A ...
I have an issue with bots crawling the mobile site and returning hits on the mobile site for desktop users. The mobile content is in /mobile/ The regular site will redirect mobile users to the ...