15

How should I interpret an empty User-agent? I have some custom analytics code and that code has to analyze only human traffic. I have got a working list of User-agents denoting human traffic and bot traffic, but the empty User-agent is proving to be problematic. And I am getting lots of traffic with the empty User-agent, about 10%.

Additionally, I have crafted the human traffic versus bot traffic user agent list by analyzing my current logs. As such I might be missing a lot of entries in there. Is there a well-maintained list of user agents denoting bot traffic or, the inverse, a list of user agents denoting human traffic?

Improve this question
5
  • 1
    The list of possible user-agents is incredibly long. For instance, look at the list of mobile-only user-agents here: zytrax.com/tech/web/mobile_ids.html
    – Hannah Vernon
    Commented Oct 12, 2012 at 11:30
  • Blank user-agent is pretty rare - what server software are you using? How are you getting the user-agent? Are you sure it really is blank, or is there some bug in your collection system that is creating blank user-agents?
    – Hannah Vernon
    Commented Oct 12, 2012 at 11:31
  • @Max - I am myself surprised at the empty user agent. I am using LAMP stack. I collect the user agent through PHP as $_SERVER['HTTP_USER_AGENT']. The code is simple; though I cannot entirely discount the possibility of user agent being there but my code failing to collect it, or database refusing to store it, I doubt that is the case.
    – Amit Agrawal
    Commented Oct 12, 2012 at 12:08
  • 1
    If you have access to Apache's access logs: Are the logged user-agents empty too?
    – f_puras
    Commented Oct 12, 2012 at 13:41
  • perhaps you have a scraper accessing your site? That might be one way visitors appear to have no USER_AGENT
    – Hannah Vernon
    Commented Oct 12, 2012 at 16:56

4 Answers 4

Reset to default
5

If you want to analyze only "human traffic" I would not count the ones with empty or missing user agent string. In my experience almost any browser will always send one. Even most privacy plugins or extensions rather fake (include other OS or Client name) or "normalize" (e.g. no release numbers) or randomize (e.g. sometimes FF, sometimes IE strings) the UA strings, but not completely remove them (as this might cause problems with some sites that rely on it, even if that's no good idea.)

A simple request with no UA can be done like this:

wget --user-agent="" www.example.com

As you see you can add anything you want. Sites that store and publish UA's found "in the wild" are not of great use as they find lot's of crap.

Maybe someone just recursively fetched your content. Or used some SEO tool to analyze your site (some allow users to manually change the header, others with the intent to ignore a robots.txt line). Things like that. In those situations UA header is often faked to hide client and purpose.

If these requests keep constantly around it might be helpful to further analyze the headers (Proxies?) or the IPs (A certain block? Privacy concerned company/Proxy?)

Improve this answer
2
  • @AaronCicali Hi! Do you have numbers or sites to back that up? There are steps removing the depth of information like described on the Chromium Blog. But it's no longer true is IMHO not quite correct.
    – initall
    Commented Dec 22, 2021 at 10:12
  • You're right! I was referencing bad data. Indeed it seems legitimate user agents do label themselves.
    – Aaron Cicali
    Commented Dec 22, 2021 at 21:42
2

I work for a security company and among other things we monitor Bad Bot traffic.

Based on my experience, humans visits with blank user-agent data indicate scraping/spamming attempts (usually scraping) made by "headless browser" bots.

These visitors can sometimes execute JS, and so they will appear in GA - still, this dose not make them human :)

Apologize for the "plug" but please know that, if needed, we offer free Bad Bot protection services - coupled with CDN acceleration and other goodies.

In this specific case our system would recognize this visit as "suspicious", verified it against known attack vectors and - if still unsure - performed further test and challenges. These challenges are performed seamlessly, without causing any delay to the session.

Improve this answer
0
0

I am seeing a few comments in the answers to this question comparing the User-Agent to hiding your identity or being human. This is an absurd comparison. User-Agent has nothing to do with identity or being human.

Think of it like footwear. You are asking your visitors what type footwear they are wearing before you are letting them in. Most common uses of this is to know what type of carpet you need to roll out, the nice red carpet for clean dress shoes, the ugly doormat for muddy boots, and no carpet for the visitors that are allergic carpets.

When the visitors don't want to say what footwear they have (aka. empty User-Agent) you ignore them.

Yes, there are plenty of good practices that try to assume things about the web request based on the User-Agent and other request header information. They might work great 99% of the time, but like with so many other similar practices they are prone false-positives and thereby harming the normal ignorant users.

Having run into the issue of accidentally using an empty User-Agent myself, I can definitely say that it is not fun when a web service treat you differently just because you didn't think to tell it about your footwear.

Improve this answer
-1

Every bit of software that accesses the internet isn't magically given a user agent. Software developers have to program that functionality into their software. Your blank user agent just means that a software developer forgot to add a user agent to their software.

Improve this answer
4
  • Or that a browser user removed/blocked the UA string.
    – unor
    Commented Oct 14, 2012 at 21:07
  • 5
    This is wrong. Saying it "just means" implies that it's usually or at least often the reason. Few people use http software who's developer would have skipped the UA out of laziness. If anything it almost always indicates the traffic source didn't want to be identified, and is deeply associated with malicious or exploitative traffic. Unfortunately some big companies (Facebook) have used empty user agent strings in the past, so it's not necessarily wise to block them entirely.
    – jerclarke
    Commented Jul 21, 2014 at 16:56
  • 1
    @jeremyclarke +1 This kind of behavior should never be overlooked! And shame on Facebook! If a legitimate bot or browser needs to look at a site they should always have an identifier. After all, they are entering someone else's property. Having no user-agent is like a burglar sneaking in with a mask on to obscure their identity.
    – whitebeard
    Commented Nov 8, 2015 at 0:30
  • 2
    It's like someone with their face covered walking back and forth across the sidwalk in front of your home. Not necessarily illegal, but if you don't answer the door they shouldn't assume you're a paranoid nut.
    – jerclarke
    Commented Nov 9, 2015 at 15:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.