Hot answers tagged web-crawlers
Are these bots that index my site(s) or are they malicious by nature and should be banned using an IP deny filter? You tell us. If they're interacting with your site as any normal user would there is no need to block them. If they're using the site more than visitors from other regions then your site may appeal to them then users from other regions. ...
Only top voted, non community-wiki answers of a minimum length are eligible