Banning unresolved IP addresses

Question

My server has run out of memory a few times in the past two days, because of which the site crashed. I checked AWstats and found that hundreds of unresolved IP addresss have been logged, and the number of hits from them seems unreasonable:

I'm going to ban the top 10 IP addresses as they all seem spammy to me. But there are other hundreds of IP addresses that seems to be causing thousands of hits, and I really don't know if they are legitimate request from users using browsers. I could ban all of them using htaccess, but that doesn't seem very practical nor is a long term solution. (The site gets around 3 million pageviews/month.)

My questions are:

1. How do I filter out legitimate users from unwanted bots or IP addresses that are scraping content?
2. Should I go ahead and ban all "unresolved IP addresses"?
3. Is there an automated way of banning spammy IP addresses?

Answer 1

BOTS, CRAWLERS and SCRAPERS use IP addresses in blocks, its rare to see one that will use the same IP address over and over. IP blocks are often sold, brought and then sold again! some of these blocks will resolve, some will not depending on the setup they use, because of this banning IP from a resolve is useless, because you risk banning real users, using VPNs, Proxies and so forth, which also may not resolve.

Banning IP addresses by self-monitoring logs is not only a waste of time but also will only work temporary until they rotate IP blocks again. The easiest way, and the best way... is to setup your server to temporarly block users/bots/crawlers whom request far to fast, similar to what Stack Exchange uses, and all the other top sites that receive insane traffic.

If you don't have the time or skills to build your own in-house blocker then your best bet would be to look at fail2ban.

Fail2Ban

Your best off asking security related questions on the Security Stack as obviously it attracts more experts in this field, here is a question and answer I found on how to setup fail2ban with a temporary block when visitors request to fast. Note, that you can also setup in additional to the temporarly block a perm, should they continue to do so whilst blocked.

fail2ban is an easy-to-implement solution in these cases. Add a block-all-dem-noobs.conf file to your filter.d directory, something like this:

[Definition]
failregex = ^ -.*GET

Translation: a RegExp to find GET requests

Then create a new entry in your jail.conf, something like this

[block-all-dem-noobs]
enabled = true
port = http,https
filter = block-all-dem-noobs
logpath = /var/log/httpd/access.log
maxretry = 100
findtime = 5
bantime = 600
action = iptables[name=HTTP, port=http, protocol=tcp]

Translation: Look through my access.log file, then block for 600 seconds (10 minutes) the IP addresses that made 100 requests in 5 seconds

One major drawback, though, is that this might produce false positives for NATed users, as they'll all appear as one IP address to you.

Answer 2

Definitely use fail2ban as Simon's answer recommends, because it protects against other issues in its default configuration. You mention the machine is running out of memory though, and fail2ban needs to monitor your logs and react to these bots, which uses memory. You may still run out of resources and if that happens you need something more efficient.

I recommend you also use iptables directly. Fail2ban uses iptables to react to attacks by dropping problematic IP addresses, but there is no reason you cannot be proactive and simply 'preload' a block of known bad IPs.

As an example, here is what I use to silently drop poneytelecom's entire range of IPs:

sudo iptables -A INPUT -s 62.210.0.0/16 -j DROP

To efficiently drop all traffic from the top 8 'greatest hits' in your screenshot you could use

sudo iptables -A INPUT -s 123.30.175.0/16 -j DROP

Two important things to remember about this though. Changes will not persist across reboots and you may block legitimate visitors coming from these IP address ranges. For the latter, you need to look up the IP addresses (try WHOIS) and make a decision whether they are an acceptable loss. For me, poneytelecom definitely was.

And to make your iptables rule(s) persistent on any Debian-based distro try the iptables-persistent package. Install and enable it with this command:

sudo aptitude install iptables-persistent && sudo service iptables-persistent start

If you change any rules later you will need to save them with sudo service iptables-persistent save. (also note iptables-persistent may be named netfilter-persistent in some distros)

Good luck and careful with iptables, my examples will work perfectly and are tested but if you are adapting and testing new ones make sure you read the man pages and understand the consequences of each new rule, or you risk making the server inaccessible to all, and you may have to re-image and start over.