3

Below is the same IP logged in at the exact same time on two different devices. I see this all through my site (with different IPs and having something to do with CFNetwork). I have blocked CFNetwork from getting anything but a 403 but what is happening with the other UA that's getting what they request? How are they doing this? Two CPUs at once? I can't figure it out? Thanks for any help.

 99.118.18.150 - - [13/May/2017:14:19:29 -0500] "GET / HTTP/1.1" 200 31268 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/603.1.30 (KHTML, like Gecko)"
 99.118.18.150 - - [13/May/2017:14:19:30 -0500] "GET /apple-touch-icon-precomposed.png HTTP/1.1" 403 1044 "-" "Safari/12603.1.30.0.34 CFNetwork/811.4.18 Darwin/16.5.0 (x86_64)"
 99.118.18.150 - - [13/May/2017:14:19:30 -0500] "GET /apple-touch-icon.png HTTP/1.1" 403 1032 "-" "Safari/12603.1.30.0.34 CFNetwork/811.4.18 Darwin/16.5.0 (x86_64)"
 99.118.18.150 - - [13/May/2017:14:19:30 -0500] "GET / HTTP/1.1" 200 31268 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 10_0 like Mac OS X) AppleWebKit/602.1.38 (KHTML, like Gecko) Version/10.0 Mobile/14A300 Safari/602.1"
 99.118.18.150 - - [13/May/2017:14:19:30 -0500] "GET /apple-touch-icon-precomposed.png HTTP/1.1" 403 1044 "-" "Safari/12603.1.30.0.34 CFNetwork/811.4.18 Darwin/16.5.0 (x86_64)"
 99.118.18.150 - - [13/May/2017:14:19:30 -0500] "GET /apple-touch-icon.png HTTP/1.1" 403 1032 "-" "Safari/12603.1.30.0.34 CFNetwork/811.4.18 Darwin/16.5.0 (x86_64)"

One other thing, in the example above with the 200; what page are they asking for (see above and below)? These have no requests that I see.

 99.118.18.150 - - [13/May/2017:14:19:29 -0500] "GET / HTTP/1.1" 200 31268 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/603.1.30 (KHTML, like Gecko) 
  • User agents are unreliable because they are so easy to fake. Block using the IP address or IP address block only. – closetnoc May 14 '17 at 1:42
  • It's clearly two bots, running from the same server. Focus on improving the content of the site, things like this are not important. – Simon Hayter May 14 '17 at 5:24
3

It's clearly two bots, running from the same server, one for desktop, one for mobile. Personally, I'd focus more on the improvement of the content than worrying about things like these, they are not important and your hosting should be good enough to handle bots. It's impossible to block them all... they just change IP address and user agent.

  • Thanks for the replies.I will continue to block the CFNetwork. They hit my site 24 hours a day. Even as they are getting a 403. Clearly a bot. – user77174 May 14 '17 at 14:58
  • 2
    Use Cloudflare if you're concerned about bots, it'll offload most of the traffic. – Simon Hayter May 14 '17 at 15:38
2

Found this (see below). So it seems to be from one device.

The Mobile Safari for example try to download the touch or favicon´s from a website

 "GET /apple-touch-icon-precomposed.png HTTP/1.1" 404 89 "-" 
 "MobileSafari/9537.53 CFNetwork/672.0.8 Darwin/14.0.0"
 "GET /apple-touch-icon.png HTTP/1.1" 404 89 "-" "MobileSafari/9537.53 CFNetwork/672.0.8 Darwin/14.0.0"
 "GET /favicon.ico HTTP/1.1" 404 89 "-" "MobileSafari/9537.53 CFNetwork/672.0.8 Darwin/14.0.0""

While the Mobile Safari retrieve the Website HTML with another User Agent String

 "GET / HTTP/1.1" 200 5474 "-" "Mozilla/5.0 (iPad; CPU OS 7_0_4 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11B554a Safari/9537.53"

Other iOS Apps did this in the same way, the Mercury Browser for example sends the same request like the Mobile Safari, with a first token Mercury/Version.

From - https://user-agents.me/cfnetwork-version-list

1

How are they doing this? Two CPUs at once?

If the confusion is how is the same IP address being used by two (or more) seemingly different devices (or servers), then this will happen for all devices behind the same NAT, from small home networks to large company office blocks. All these devices share the same external IP address, but communicate on a different port. The remote port does not seem to be shown in your access log, but this can be enabled with the %{remote}p format string if desired.

in the example above with the 200; what page are they asking for? These have no requests that I see.

The log entry shows the first line of the request headers:

"GET / HTTP/1.1"

This contains the URL-path - a single slash (/). ie. http://example.com/ (the document root).

0

"GET / HTTP/1.1" 200 means they are trying to get your website home first. Probably just to check if site is reachable. A bot can easily change user-agent using PHP Curl or another library. Once it sees your site is reachable, it tries to access "apple-touch" image files. An HTML5 Boilerplate site will have these images in the home page. So the bot tries to identify if your site has boilerplate. Maybe the person running the bot knows some exploit for boilerplate sites.

I had the same thing on my site. I didn't have any images with "apple-touch" prefix on my site. So, in cloudflare I made a new firewall rule. And selected "URI Full" "contains" "apple-touch" Block.

These bots can be dangerous. So best just to block them using URI method. Because if we block IP or user-agent, they might just keep changing them.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.