7

My website, which typically has very low volume, suddenly received a huge spike in traffic all from the same user agent. We were receiving about 3 requests per second for a sustained period of about an hour all from the user agent "Facebot Twitterbot." The requests all look like this, to one URI:

Dec 16 00:35:36 website app/web.1 10.1.52.64 - - [16/Dec/2021:02:35:36 -0600] "GET /signup/ HTTP/1.1" 200 2299 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0"

We had a similar traffic event to another page in the past that lasted a bit longer about two weeks ago. This isn't a huge problem, since the site can handle the traffic and it seems to end after some time, but it does slow the response time by a few milliseconds typically and it spams our logs and makes it difficult to debug actual problems that crop up.

I have looked into this user agent a bit (see: here and here) it seems to be an IOS imessage utility function, though it it still unclear what exactly causes it. Is this traffic normal, should I have any reason to be concerned?

3
  • 1
    I am also seeing this, confused, but it's coming from my own home IP address. I see 16 requests with this user agent, all at the same second, all getting 401 "unauthorized" responses. I noticed because fail2ban suddenly banned my own IP address over it! The requested pages are /, /favicon.ico, /apple-touch-icon.png, and /apple-touch-icon-precomposed.png. At the same moment, there were 3 requests from another user agent that seems like my usual iPhone/safari user agent.
    – dcc310
    Dec 28, 2021 at 21:12
  • 1
    This post might be interesting: medium.com/@siggi/…. I am only seeing this stuff in my logs from the past 2 days (and just 2 requests 2 nights ago), and I have 10 days of logs. I think I only mentioned this domain in iMessage about a month ago when I bought it, so if it's from iMessage previews, it must be reprocessing very old previews...
    – dcc310
    Dec 28, 2021 at 22:07
  • @dcc310 Thank you! This indeed does seem to be the cause for us, though the volume of requests is still concerning.
    – Wold
    Jan 19 at 6:06

1 Answer 1

5

This is the iMessages app's crawler (from the users phone itself, not a Apple server).

Apple has chosen to use this useragent in their iMessages app to ensure the unfurling of the URL and the rich preview works more often than not. When someone sends a URL, their phone attempts to retrieve a picture from the website to use in the rich preview. They are assuming that the Facebook and Twitter crawlers are whitelisted by most websites, so they're abusing that to ensure the crawler's requests are not blocked.

The most common requests from this useragent are:

  • /favicon.ico
  • /apple-touch-icon.png
  • /apple-touch-icon-precomposed.png

It's bizarre because no one blocks standard iPhone/Safari useragents, so the addition of facebookexternalhit/1.1 Facebot Twitterbot/1.0 doesn't really make sense to me.

It may also be Facebook messenger or a Facebook bot, but I haven't seen any real evidence for that except for some random posts online.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.