Tagged Questions
22
votes
1answer
3k views
Bingbot request for trafficbasedsspsitemap.xml which does not exist
The logs for a website I manage show a request for a non-existent file by Bingbot. The details of the request are
Path: /trafficbasedsspsitemap.xml
Useragent: "Mozilla/5.0 (compatible; bingbot/2.0; ...
3
votes
4answers
1k views
Why deny access to website for msnbot/bingbot?
I've seen quite a lot of tutorials that recommend you to ban user agents containing the strings libwww-perl and msnbot. I understand why one would ban libwww-perl, it's mainly if not only used for ...
3
votes
1answer
478 views
Why is BingBot causing so many 404 errors by removing letters from url's?
Since the ForceRecrawl problem went away Bing has come back with some new trends.
I'm seeing many URLs missing the last letter, or missing a few letters, and some other wired guess URLs. It also ...
1
vote
1answer
656 views
MSNBot/BingBot not reporting it is a bot?
I have apache logs from my server, and I filter out visits from bots/crawlers/scrapers using a python script that checks for user agent strings containing text like 'bot', 'googlebot', etc.
Lately, ...
