2
votes
1answer
166 views

Is there a reason msnbot crawls in spikes?

I've been experiencing high RPM spikes recently. Something like this: When I debugged, I've found reasons to believe the reason is the msnbot suddenly makes a massive crawl and then stops. I assume ...
0
votes
0answers
58 views

Bingbot requesting incorrect robots.txt path

A .NET site I am currently working on is being hit rather frequently by the Bingbot on the path: /:80/robots.txt This is obviously throwing the following exception: System.Web.HttpException: A ...
2
votes
2answers
758 views

Prevent crawling of mobile site

I have an issue with bots crawling the mobile site and returning hits on the mobile site for desktop users. The mobile content is in /mobile/ The regular site will redirect mobile users to the ...