How do I capture incoming HTTP requests for analysis?

Question

I have a site that is hit pretty hard by a 'bot script that thinks it is possible to brute-force the Admin password by "trying" out different passwords with the username "Admin". Here is an excerpt from my log:

uname   password     IP-address
-------+--------+-------------------
'Admin' '300283' IP: 220.250.61.78
'Admin' '300284' IP: 94.255.35.226
'Admin' '300281' IP: 176.108.152.178
'Admin' '300191' IP: 109.162.54.139
'Admin' '300280' IP: 213.87.138.83
'Admin' '300193' IP: 2.242.0.165

A typical request from the bot is logged like this in Apache's access_log and since it is POST the interesting bit (i.e. that it tries to login on the admin user) is not in the header.

 109.162.54.139 - - [08/Sep/2013:17:52:44 +0200] "POST /user HTTP/1.1" 200 20571 "http://example.org/user" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72 Safari/537.36"

The 'bot does about 8 login request per second, and it has been doing this for four days so far.

The site attacked is owned by a completely uncontroversial NGO. I run it pro bono on a small shared host. The configuration doesn't really scale well for this sort of thing, so performance suffers. It is obviously a pretty stupid script, because when I set up Apache to reply 403 to every request from everone except from my IP, it just kept going on as nothing had changed. (Doing so restored performance, so if I can get this blocked by Apache, I've won.)

As you can see from the log excerpt, whoever's doing this commands a botnet with a vast number of IP-addresses. Going through the logs, I notice that it very seldom reuses an IP-address, so blocking the IP-adresses, as suggested in the upvoted answer to this question: Tactics for dealing with misbehaving robots is not an option.

Btw: there is no account with username "Admin" on the system, nor is there any sensitive or valuable information on the system, so it puzzles me why anyone is doing this.

One thing I want to try out is to analyzing the incoming requests, I hope I shall be able discover some sort of pattern. And when I've found a pattern, set up some rule in Apache to block requests matching the pattern. But I have no idea how to capture the incoming requests for analysis, so my question is: How do I capture the POST part of incoming HTTP requests for analysis?

Answer 1

How do I capture the POST part of incoming HTTP requests for analysis?

Try using ModSecurity - from its documentation on HTTP Traffic Logging:

Web servers are typically well-equipped to log traffic in a form useful for marketing analyses, but fall short logging traffic to web applications. In particular, most are not capable of logging the request bodies. Your adversaries know this, and that is why most attacks are now carried out via POST requests, rendering your systems blind. ModSecurity makes full HTTP transaction logging possible, allowing complete requests and responses to be logged. Its logging facilities also allow fine-grained decisions to be made about exactly what is logged and when, ensuring only the relevant data is recorded.

Using ARGS_POST as covered under ARGS should allow you to capture the POST part of incoming HTTP requests for analysis:

ARGS is a collection and can be used on its own (means all arguments including the POST Payload), with a static parameter (matches arguments with that name), or with a regular expression (matches all arguments with name that matches the regular expression). To look at only the query string or body arguments, see the ARGS_GET and ARGS_POST collections.

See the other ModSecurity features here, like: real-time monitoring, attack detection, and attack prevention...

---

...whoever's doing this commands a botnet with a vast number of IP-addresses. Going through the logs, I notice that it very seldom reuses an IP-address

Malicious bots can connect through open proxies all over the world (typically without permission), each having different IP addresses. They can also use trojan horses and viruses on hijacked client computers/servers to do their work. So you are correct that blocking IP addresses is not an option to prevent these, though there are honey pot projects like this one which attempt to provide a database of abusive IP addresses.