I am all for allowing any legitimate search engines to visit my site, but I've noticed that on my business-card-style website about every other request comes from MJ12Bot, yet for well-known reasons of them being a niche SEO bot, they don't even actually send any human visitors back, so, I'm quite disappointed about the noise they generate.
% cut -f12- -d" " constantine.su.access.log | sort | uniq -c | fgrep -i -e bot -e spider | sort -nr | head
421 "Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)"
69 "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
64 "woobot/1.1"
62 "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
61 "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
39 "Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)"
30 "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
14 "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
13 "woobot/2.0"
12 "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
Is there a way to quiet down MJ12Bot ambitions (by something like 20×)? Or, due to the distributed nature of the MJ12bot project, do I just have to block 'em all outright as parasitic?
cut -f12- -d" " constantine.su.access.log | sort | uniq -c | fgrep -i -e bot -e spider | sort -nr | head
on the server.