2

My server has been crashes several times the last 24 hours by a new extremely aggressive hostile distributed web crawler.

It is hitting over 1000 pages per minute and seems to be crawling all content on our website repeatedly. It ignores robots.txt and is disguising itself with random IP addresses and agent strings.

I verified it is a crawler, not real traffic as the hits do not show up in Google Analytics, and are following a web crawler pattern of hitting every page (our site has a lot of content).

I have noticed it only the last 24 hours, and on several of our websites, but the main issue is on our site with the most content. It does not seem to be content in crawling the site once, but is hitting the same pages repeatedly. Not sure of its purpose, but it is very hostile and aggressive, and because the IP addresses and agent string seems to always be different, I don't see any way to block it.

I tried to block all Chine IP address using iptables and, https://askubuntu.com/questions/868334/block-china-with-iptables

that seemed to help significantly, but this does not seem to block all China IPs as I still see some, and the crawler seems to also have many non-China IPs (although the majority are China). Given the number of IPs it could be some sort of worm intent on bringing down the entire Internet...

Some of the IPs and agent strings are,

39.181.228.211 - Zhejiang

183.27.48.119 - Shanxi

171.116.206.136 - Guangdong

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World)

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0

Has anyone else seen this new crawler on their websites?

Any idea of how to stop it?

Anyone know what it is, or what its purpose is, where it originates from?

Improve this question
4
  • So I was hoping this bot would go away, but after 5 days it is still hitting our website >1 million times per day, even with all of China blocked in our Firewall (at least most of China), my guess is this is maybe from Tencent? Some new distributed crawler, someone with an insane amount of resources at any rate.
    – James
    Feb 28, 2020 at 19:43
  • After 10 days it is still hitting our server >1 million times per day. My current guess is it is using Tor to randomize its IP, or some kind of worm that has infected a massive amount of servers.
    – James
    Mar 4, 2020 at 18:07
  • The bot finally stopped hitting our website the other day. Hopefully it does not come back.
    – James
    Mar 10, 2020 at 11:50
  • The bot is back, starting yesterday (05-13-2020) not sure if anyone else has scene this yet
    – James
    May 14, 2020 at 16:42

1 Answer 1

Reset to default
2

I recommended you use WAF(Web Application Firewall) like Cloudflare. Cloudflare WAF got more than 145 rules to protect from almost all types of web application attacks.

For example use:

You can clean up bad bots https://blog.cloudflare.com/cleaning-up-bad-bots/

You can identify and mitigate scrapers and automated requests. https://support.cloudflare.com/hc/en-us/articles/360027519452-Understanding-Cloudflare-Bot-Management

You can block the county using Cloudflare easily: https://stackoverflow.com/questions/34560058/how-to-block-countries-from-server-when-using-cloudflare

You can block by a user agent https://support.cloudflare.com/hc/en-us/articles/115001856951-Understanding-Cloudflare-User-Agent-Blocking

If Cloudflare detect bad visitor/bot it's will automatically show captcha https://support.cloudflare.com/hc/en-us/articles/200170136-Understanding-Cloudflare-Challenge-Passage-Captcha-

other Cloudflare alternatives:

  • Sucuri
  • StackPath
  • WebARX
  • etc
Improve this answer
5
  • thanks, but not sure if it would be able to block this, as the IP are random, and the agent string is random, can't see how it would differentiate from normal traffic. Also I would prefer to not have to block all of China, as we have users in China, and this crawler has many IPs outside of China as well.
    – James
    Feb 24, 2020 at 21:39
  • using WAF not only can block the attacker but also detect and prevent the robot user. check it out: support.cloudflare.com/hc/en-us/articles/… blog.cloudflare.com/cleaning-up-bad-bots
    – Muhammad Dyas Yaskur
    Feb 24, 2020 at 23:16
  • I'll see if I can test this, seems to have a free trial, then $200/m which is pretty expensive
    – James
    Feb 28, 2020 at 19:45
  • No, they have a free version which still good enough. Challenge Passage is a feature that prevents a bad visitor and available on the free subscription.
    – Muhammad Dyas Yaskur
    Feb 29, 2020 at 4:58
  • I have setup Cloudflare. I setup a firewall to block china, and the MSIE agent, but it did not help. Instead of getting 1 million hits from China, I get 1 million hits from every other country, instead of 1 million hits with MSIE agent, it uses other agents, so now I get 4 million hits with 3 million blocked...
    – James
    May 19, 2020 at 14:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.