6

I am all for allowing any legitimate search engines to visit my site, but I've noticed that on my business-card-style website about every other request comes from MJ12Bot, yet for well-known reasons of them being a niche SEO bot, they don't even actually send any human visitors back, so, I'm quite disappointed about the noise they generate.

% cut -f12- -d" " constantine.su.access.log | sort | uniq -c | fgrep -i -e bot -e spider | sort -nr | head
 421 "Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)"
  69 "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
  64 "woobot/1.1"
  62 "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
  61 "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
  39 "Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)"
  30 "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
  14 "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
  13 "woobot/2.0"
  12 "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

Is there a way to quiet down MJ12Bot ambitions (by something like 20×)? Or, due to the distributed nature of the MJ12bot project, do I just have to block 'em all outright as parasitic?

|improve this question
  • I'm looking at your log in the question and it doesn't show any one computer spamming your server. MJ12Bot appeared only once. – Mike Apr 24 '17 at 18:54
  • @Mike, can you elaborate, please? which log are you looking at? what does that big number in front of the MJ12Bot line means? – cnst Apr 24 '17 at 21:25
  • your log that printed on your screen that you pasted in your question when you executed cut -f12- -d" " constantine.su.access.log | sort | uniq -c | fgrep -i -e bot -e spider | sort -nr | head on the server. – Mike Apr 25 '17 at 1:23
  • and the big number based on the linux command likely means how many times an entry was found in the log file. – Mike Apr 25 '17 at 1:24
  • @Mike, so, how did MJ12Bot appeared only once? – cnst Apr 25 '17 at 9:10
2

Is there a way to quiet down MJ12Bot ambitions

The MJ12Bot reportedly obeys robots.txt and the (non-standard) Crawl-Delay directive:

How can I slow down MJ12bot?

You can easily slow down bot by adding the following to your robots.txt file:

User-Agent: MJ12bot
Crawl-Delay: 5

Crawl-Delay should be an integer number and it signifies number of seconds of wait between requests. MJ12bot will make an up to 20 seconds delay between requests to your site - note however that while it is unlikely, it is still possible your site may have been crawled from multiple MJ12bots at the same time. Making high Crawl-Delay should minimise impact on your site.

Reference:
http://mj12bot.com/

|improve this answer
  • Crawl-Delay is not an answer, because with only a couple of pages on the site, it will have zero effect on the total number of requests per month. – cnst May 25 '16 at 23:06
  • 4
    Only two pages!? Is this excessive crawling causing issues for your site? What period does the log summary in your question cover? Do these pages change regularly? Are you setting appropriate cache headers (although whether the MJ12Bot will honour these is another matter)? – MrWhite May 25 '16 at 23:46
  • 1
    I have seen this in a while, however, it used to be that scrapers would use the MJ12bot agent name to disguise themselves. I am not sure this is happening anymore, but it is something to look at. Otherwise, as you said, MJ12bot behaves itself very well. Cheers!! – closetnoc May 25 '16 at 23:51
  • The logs cover 25 days, the pages are static, change infrequently and have default headers in nginx. Basically, I don't mind if MJ12bot will visit me a couple of times per year, but being every other visitor is just way too much. I think the issue may be due to the fact that MJ12bot is not centralised, so, one bot has no idea that another one has already been there. – cnst May 26 '16 at 0:26
2

From your comments on another answer, MJ12Bot is visiting your site less than once an hour (421 times in 25 days.) The best thing to do is to not worry about it. Crawl-Delay is useless for you because no crawler will obey a craw-delay so large.

|improve this answer
2

MJ12bot adheres to the robots.txt standard. If you want the bot to prevent website from being crawled then add the following text to your robots.txt:

User-agent: MJ12bot
Disallow: /
|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.