1

A site I help take care of has just started getting crawled by somebody from 172.31.13.241 and 172.31.42.227 . According to WHOIS info, these are part of a block reserved by IANA for special purposes, specifically private networks.

They're making 400 rpm - while not too bad, and not a problem for the infrastructure, it doesn't entirely feel like benign crawling (e.g. Googlebot will make far fewer requests it seems, and I've previously had other sites crawled at 1 to 5 rpm).

I've banned them for the moment, but would like to know if there is a plausible explanation which points to private IPs in Apache access logs being a normal, "usual" thing for some reason. So far I've been unable to find any such explanations, in Superuser.com, Webmasters SE, or the wider net.

EDIT: Thanks for the comprehensive overview John McNamara. The 2 IPs turned out to belong to the Elastic Load Balancer in front of our AWS instances. The thing is, it usually passes on the IP of the request directly to the instances (e.g. I can see GoogleBot's IP, I can see users' and my own IP while browsing). I used the load balancer's logs to find out the public IP, it turned out to be a product we are trialling out which requires crawling. I believe they usually crawl much bigger websites than ours, hence the unusually high number.

1

Is there anything unusual or nefarious about private IP addresses in server logs?

would like to know if there is a plausible explanation which points to private IPs in Apache access logs being a normal, "usual" thing for some reason

Unusual, no. Nefarious, possibly, it really depends on what the traffic is attempting to do.

from https://www.iana.org/help/abuse-answers

If you see an apparent attack, or spam, coming from one of these address ranges, then either it is coming from your local environment, your ISP, or the address has been "spoofed".

The root cause could be any of the following

  1. misconfigured device on your LAN causing many requests
  2. misconfigured device on your ISP's network causing many requests
  3. malware/active attacker on your LAN
  4. malware/active attacker on your ISP's network
  5. active attacker in internet spoofing IP addresses
  • The statement coming from your local environment, your ISP, or the address has been "spoofed" is not necessarily true. Unfortunately, a ton of routers route packets in either direction that are not supposed to be routable. This problem is so pervasive, that it is possible to route private IP addresses through the Internet. However, short of the local network, it is still extremely unlikely that a private IP is coming from the Internet without being spoofed. For this reason, regardless of the access, it is wise to always block private IP addresses for security reasons. – closetnoc May 28 '15 at 21:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.