My server has been crashes several times the last 24 hours by a new extremely aggressive hostile distributed web crawler.
It is hitting over 1000 pages per minute and seems to be crawling all content on our website repeatedly. It ignores robots.txt and is disguising itself with random IP addresses and agent strings.
I verified it is a crawler, not real traffic as the hits do not show up in Google Analytics, and are following a web crawler pattern of hitting every page (our site has a lot of content).
I have noticed it only the last 24 hours, and on several of our websites, but the main issue is on our site with the most content. It does not seem to be content in crawling the site once, but is hitting the same pages repeatedly. Not sure of its purpose, but it is very hostile and aggressive, and because the IP addresses and agent string seems to always be different, I don't see any way to block it.
I tried to block all Chine IP address using iptables and, https://askubuntu.com/questions/868334/block-china-with-iptables
that seemed to help significantly, but this does not seem to block all China IPs as I still see some, and the crawler seems to also have many non-China IPs (although the majority are China). Given the number of IPs it could be some sort of worm intent on bringing down the entire Internet...
Some of the IPs and agent strings are,
39.181.228.211 - Zhejiang
183.27.48.119 - Shanxi
171.116.206.136 - Guangdong
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0
Has anyone else seen this new crawler on their websites?
Any idea of how to stop it?
Anyone know what it is, or what its purpose is, where it originates from?