Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

How should I interpret a empty User-agent? I have some custom analytics code and that code has to analyze only human traffic. I have got a working list of User-agents denoting human traffic, and bot traffic, but the empty User-agent is proving to be problematic. And I am getting lots of traffic with empty user agent - 10%.

Additionally - I have crafted the human traffic versus bot traffic user agent list by analyzing my current logs. As such I might be missing a lot of entries in there. Is there a well maintained list of user agents denoting bot traffic, OR the inverse a list of user agents denoting human traffic?

share|improve this question

migrated from stackoverflow.com Oct 13 '12 at 21:35

This question came from our site for professional and enthusiast programmers.

1  
The list of possible user-agents is incredibly long. For instance, look at the list of mobile-only user-agents here: zytrax.com/tech/web/mobile_ids.html –  Max Vernon Oct 12 '12 at 11:30
    
Blank user-agent is pretty rare - what server software are you using? How are you getting the user-agent? Are you sure it really is blank, or is there some bug in your collection system that is creating blank user-agents? –  Max Vernon Oct 12 '12 at 11:31
    
@Max - I am myself surprised at the empty user agent. I am using LAMP stack. I collect the user agent through PHP as $_SERVER['HTTP_USER_AGENT']. The code is simple; though I cannot entirely discount the possibility of user agent being there but my code failing to collect it, or database refusing to store it, I doubt that is the case. –  Amit Agrawal Oct 12 '12 at 12:08
1  
If you have access to Apache's access logs: Are the logged user-agents empty too? –  f_puras Oct 12 '12 at 13:41
    
perhaps you have a scraper accessing your site? That might be one way visitors appear to have no USER_AGENT –  Max Vernon Oct 12 '12 at 16:56
add comment

3 Answers 3

up vote 2 down vote accepted

If you want to analyze only "human traffic" I would not count the ones with empty or missing user agent string. In my experience almost any browser will always send one. Even most privacy plugins or extensions rather fake (include other OS or Client name) or "normalize" (e.g. no release numbers) or randomize (e.g. sometimes FF, sometimes IE strings) the UA strings, but not completely remove them (as this might cause problems with some sites that rely on it, even if that's no good idea.)

A simple request with no UA can be done like this:

wget --user-agent="" www.example.com

As you see you can add anything you want. Sites that store and publish UA's found "in the wild" are not of great use as they find lot's of crap.

Maybe someone just recursively fetched your content. Or used some SEO tool to analyze your site (some allow users to manually change the header, others with the intent to ignore a robots.txt line). Things like that. In those situations UA header is often faked to hide client and purpose.

If these requests keep constantly around it might be helpful to further analyze the headers (Proxies?) or the IPs (A certain block? Privacy concerned company/Proxy?)

share|improve this answer
add comment

I work for a security company and among other things we monitor Bad Bot traffic.

Based on my experience, humans visits with blank user-agent data indicate scraping/spamming attempts (usually scraping) made by "headless browser" bots.

These visitors can sometimes execute JS, and so they will appear in GA - still, this dose not make them human :)

Apologize for the "plug" but please know that, if needed, we offer free Bad Bot protection services - coupled with CDN acceleration and other goodies.

In this specific case our system would recognize this visit as "suspicious", verified it against known attack vectors and - if still unsure - performed further test and challenges. These challenges are performed seamlessly, without causing any delay to the session.

share|improve this answer
add comment

Every bit of software that accesses the internet isn't magically given a user agent. Software developers have to program that functionality into their software. Your blank user agent just means that a software developer forgot to add a user agent to their software.

share|improve this answer
    
Or that a browser user removed/blocked the UA string. –  unor Oct 14 '12 at 21:07
    
This is wrong. Saying it "just means" implies that it's usually or at least often the reason. Few people use http software who's developer would have skipped the UA out of laziness. If anything it almost always indicates the traffic source didn't want to be identified, and is deeply associated with malicious or exploitative traffic. Unfortunately some big companies (Facebook) have used empty user agent strings in the past, so it's not necessarily wise to block them entirely. –  jeremyclarke yesterday
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.