Hot answers tagged web-crawlers
Baidu spider also does not properly obey robots.txt in some cases. They seem to have a history of doing things their own (wrong) way. Unless you have a Chinese website that gets a significant amount of traffic from the Baidu search engine, why would you keep old server up just to make the Baidu spider happy?
Only top voted, non community-wiki answers of a minimum length are eligible