7

My server has run out of memory a few times in the past two days, because of which the site crashed. I checked AWstats and found that hundreds of unresolved IP addresss have been logged, and the number of hits from them seems unreasonable:

Enter image description here

I'm going to ban the top 10 IP addresses as they all seem spammy to me. But there are other hundreds of IP addresses that seems to be causing thousands of hits, and I really don't know if they are legitimate request from users using browsers. I could ban all of them using htaccess, but that doesn't seem very practical nor is a long term solution. (The site gets around 3 million pageviews/month.)

My questions are:

  1. How do I filter out legitimate users from unwanted bots or IP addresses that are scraping content?
  2. Should I go ahead and ban all "unresolved IP addresses"?
  3. Is there an automated way of banning spammy IP addresses?
  • 4
    You're fixing the wrong problem. What you should be trying to fix is why is the site running out of memory? – Josh Nov 15 '15 at 16:52
  • 1
    Is the screenshot an example of these "unresolved IPs"? Because they are not unresolved because they are unresolvable. It would seem that AWStats is simply not configured to resolve these IP addresses? – MrWhite Nov 15 '15 at 17:44
  • @Josh Fair enough. This was the first thing I looked at in the process of finding the memory problem. And I'm not able to find anything yet since the crashes seems so random. I'm not really good at server side admin things and am learning. Any suggestions? – Yeti Nov 16 '15 at 2:44
  • 1
    Sure, and sorry if my comment seemed harsh. You might want to start a new question detailing the specific symptoms of memory exhaustion you're seeing, and your configuration (php fpm, mod php, httpd.conf, etc) as there will likely be a lot of back and forth communications – Josh Nov 16 '15 at 2:49
8

BOTS, CRAWLERS and SCRAPERS use IP addresses in blocks, its rare to see one that will use the same IP address over and over. IP blocks are often sold, brought and then sold again! some of these blocks will resolve, some will not depending on the setup they use, because of this banning IP from a resolve is useless, because you risk banning real users, using VPNs, Proxies and so forth, which also may not resolve.

Banning IP addresses by self-monitoring logs is not only a waste of time but also will only work temporary until they rotate IP blocks again. The easiest way, and the best way... is to setup your server to temporarly block users/bots/crawlers whom request far to fast, similar to what Stack Exchange uses, and all the other top sites that receive insane traffic.

If you don't have the time or skills to build your own in-house blocker then your best bet would be to look at fail2ban.

Fail2Ban

Your best off asking security related questions on the Security Stack as obviously it attracts more experts in this field, here is a question and answer I found on how to setup fail2ban with a temporary block when visitors request to fast. Note, that you can also setup in additional to the temporarly block a perm, should they continue to do so whilst blocked.

fail2ban is an easy-to-implement solution in these cases. Add a block-all-dem-noobs.conf file to your filter.d directory, something like this:

[Definition]
failregex = ^ -.*GET

Translation: a RegExp to find GET requests

Then create a new entry in your jail.conf, something like this

[block-all-dem-noobs]
enabled = true
port = http,https
filter = block-all-dem-noobs
logpath = /var/log/httpd/access.log
maxretry = 100
findtime = 5
bantime = 600
action = iptables[name=HTTP, port=http, protocol=tcp]

Translation: Look through my access.log file, then block for 600 seconds (10 minutes) the IP addresses that made 100 requests in 5 seconds

One major drawback, though, is that this might produce false positives for NATed users, as they'll all appear as one IP address to you.

  • Thanks for the answer. I'll check out fail2ban. Will this affect Googlebot though? Googlebot makes ~1 million requests/day on the site! – Yeti Nov 15 '15 at 14:48
  • 2
    Google, doesn't hammer your site, it has timed delays. Your goal should be to limit the amount of server-side requests per a second, this will ultimately save you bandwidth and resources. If nasty bots want to scrap your site then it'll take them a long time to do so, Google on the other hand, has been crawling your site for a long time and doesn't require to hammer to get a big chunk at once, but you can always add Google/Bing to white list IP's but your have to look into that, personally, I just update the hosting, and ban repeat offenders automatically. – Simon Hayter Nov 15 '15 at 17:25
1

Definitely use fail2ban as Simon's answer recommends, because it protects against other issues in its default configuration. You mention the machine is running out of memory though, and fail2ban needs to monitor your logs and react to these bots, which uses memory. You may still run out of resources and if that happens you need something more efficient.

I recommend you also use iptables directly. Fail2ban uses iptables to react to attacks by dropping problematic IP addresses, but there is no reason you cannot be proactive and simply 'preload' a block of known bad IPs.

As an example, here is what I use to silently drop poneytelecom's entire range of IPs:

sudo iptables -A INPUT -s 62.210.0.0/16 -j DROP

To efficiently drop all traffic from the top 8 'greatest hits' in your screenshot you could use

sudo iptables -A INPUT -s 123.30.175.0/16 -j DROP

Two important things to remember about this though. Changes will not persist across reboots and you may block legitimate visitors coming from these IP address ranges. For the latter, you need to look up the IP addresses (try WHOIS) and make a decision whether they are an acceptable loss. For me, poneytelecom definitely was.

And to make your iptables rule(s) persistent on any Debian-based distro try the iptables-persistent package. Install and enable it with this command:

sudo aptitude install iptables-persistent && sudo service iptables-persistent start

If you change any rules later you will need to save them with sudo service iptables-persistent save. (also note iptables-persistent may be named netfilter-persistent in some distros)

Good luck and careful with iptables, my examples will work perfectly and are tested but if you are adapting and testing new ones make sure you read the man pages and understand the consequences of each new rule, or you risk making the server inaccessible to all, and you may have to re-image and start over.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.