An http header field that identifies the browser and operating system.

learn more… | top users | synonyms

3
votes
2answers
2k views

What browser is the “Apache-HttpClient/UNAVAILABLE (java 1.4)” user agent?

Some of my users are logging in with this browser: Apache-HttpClient/UNAVAILABLE (java 1.4). What browser is this?
0
votes
2answers
64 views

Googlebot-Mobile ignoring robots.txt, pretending to be Googlebot

I'm very disappointed that Googlebot is apparently ignoring my robots.txt. I have the following sole entry for my /robots.txt: location = /robots.txt { return 200 "## $host ## \n #Dear ...
8
votes
3answers
174 views

User-Agent with base64-encoded component?

(Bounty question at the bottom) I'm having an issue with a client accessing our site, and the root cause is that the WAF (Web Application Firewall) doesn't like their User-Agent string: User-Agent: ...
4
votes
1answer
57 views

User Agents with Hash

104.245.34.141 - - [09/Apr/2016:11:51:35 -0400] "GET /22327 HTTP/1.1" 200 3516 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0#19d5fd" 86.105.50.203 - - [09/Apr/2016:...
3
votes
3answers
89 views

How to determine if a user agent string has proper syntax or might be a hacking attempt?

I was checking my server via awstats to see who was visiting my site and I have a user-agent of the following value: }__test|O:21:\ Some research has lead me to believe that someone was trying to ...
0
votes
2answers
1k views

BingPreview user agent

I'm getting several invalid requests from a "BingPreview" user agent. The complete user agent header is: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b The ...
2
votes
1answer
30 views

Applications that meet the definition of a User Agent under the W3C

I'm reading through the W3C's Definition of User Agent: Definition: A user agent is any software that retrieves, renders and facilitates end user interaction with Web content, or whose user ...
7
votes
3answers
5k views

Interpretation of empty User-agent

How should I interpret a empty User-agent? I have some custom analytics code and that code has to analyze only human traffic. I have got a working list of User-agents denoting human traffic, and bot ...
5
votes
4answers
2k views

What does it mean when a User-Agent has another User-Agent inside it?

Basically, sometimes the user-agent will have its normal user-agent displayed, then at the end it will have the "User-Agent: " tag displayed, and right after it another user-agent is shown. Sometimes, ...
0
votes
1answer
32 views

Can't get my browser to show me the mobile version of my website

I'm trying to debug something in the mobile version of my website, but I just can't make it show up on my desktop browser. I'm using the plugin "User Agent Switcher" on Firefox with this user agent: ...
2
votes
1answer
48 views

Yandex mobile services robot - is it really a mobile user agent?

As you can tell, I want my website indexed properly in as many search engines as possible. I'm looking at Yandex and using its "Server Response Check" tool which is similar to "Fetch as google" tool ...
1
vote
1answer
33 views

Handling visitors computers that don't report languages or countries

I added google analytics to the mobile version of my site and three days later it seems that most of the visitors either do not have a language set or a country code set. Should I assume every ...
2
votes
3answers
392 views

How to block a user-agent that has a spacing in its name?

I got a hit from a crawler with a user-agent called DV CRAWLER which is an abvious a spam-bot. I tried to block it in both .htacess and nginx configuration as I'm running nginx as a reverse proxy in ...
0
votes
1answer
97 views

How to find out which links of website posted on Twitter

I recenlty noticed many requests from Twitterbot user agent in my website's logs which I believe it doesn't crawl by itself unless someone is posting links of my website on Twitter. These posts might ...
1
vote
2answers
505 views

What browser is sending user agent beginning mozilla/5.0+, tramslates & into &

We've got a website which has been running for a few years now. One of our customers has just started having an intermittent problem. Looking at our iis6.0 logs the service works correctly when they ...
3
votes
1answer
504 views

In Webtrends, what browser is “Mozilla <Version Unknown>”?

I am getting a ton of visits from a browser that is showing up in my Webtrends stats as "Mozilla" with version listed as "Version Unknown." It's almost 14% of my page views. This is separate from ...
1
vote
2answers
183 views

Is it okay to block all user agents that start with “PHP/5.”?

I'm getting a lot of spambots from China (and elsewhere, likely botnets) accessing my blog which have a user agent string starting with PHP/5.. These include such nonsense values as PHP/5.3.70 and PHP/...
1
vote
1answer
3k views

Detecting Android tablets vs phones with User Agent

I inherited a site that has PHP code that decides to either serve the dedicated mobile site or desktop site to the user. I would prefer that tablets get served the mobile site. Currently Android ...
4
votes
1answer
349 views

Can I block requests that have no user agent because they cause 404 errors?

Is it OK to block requests that contain no user agent? I am getting random 404 errors on URLs I have never used. These requests have no user agent. If this is the case, will it affect any of my SEO?...
2
votes
2answers
1k views

Finding different Windows XP Service Packs via User-Agent string

I am planning to get rid of the SHA1 certificate signing algorithm in our certificates and switch over to SHA-256. Problem is, that Windows XP (before SP3) is not supporting this. Therefore I need to ...
6
votes
3answers
623 views

Does the user agent in any regular browser contain 'bot' or 'crawl'?

Does the user agent in any regular browser contain 'bot' or 'crawl'? I check the user agent on my site to see if it is coming from a bot or not. If it is, I can do some little optimizations since ...
0
votes
1answer
153 views

What percent of all active IP + User Agent combinations are not unique?

It's fairly common practice to define a unique user by a 1-to-1 hash of IP + User Agent. But not all unique users have a unique IP + User Agent combination. Thus: collisions - events misattributed to ...
8
votes
4answers
15k views

Mobile phone detection (brand, model, browser etc)

What do you use to detect visitor's mobile phone, down to the model if possible? Currently we maintain our own database but it's really getting behind due to lack of manpower to maintain it, so we ...
8
votes
1answer
211 views

Combine user-agents in robots.txt

Can user-agents be listed together, followed by their common rules like so in a robots.txt? User-agent: Googlebot User-agent: ia_archiver #Alexa User-agent: BingPreview #Microsoft User-agent: bingbot ...
2
votes
2answers
1k views

How to block user agent from accessing my WordPress site with htaccess?

I have noticed large amount of traffic coming to my site from different IP's with user agent. Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/) I don't have much experience ...
1
vote
1answer
956 views

Identify this user agent - claims to be IE7 on Windows 7

We have a tracking pixel in one of our emails and our the most frequent user agent is this strange one: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0....
1
vote
2answers
70 views

How can we find real source of suspected user entries?

I have a website where I keep logging the user entries. I found some anonymous entries in my table and I suspect they are using some kind of proxy switchers. How can I find the real source of these ...
2
votes
3answers
585 views

Can web crawlers that visit my site fake their user agent?

I want to write a robots.txt file for my website and allow the famous bots (Google, Bing, and Yahoo) to crawl my website, but deny the rest. I want to know if I add User-agent: Googlebot, will fake ...
12
votes
3answers
6k views

What user agent should I set?

There is Ask bot, which sets this header: Mozilla/2.0 (compatible; Ask Jeeves/Teoma) Considering this, I have the following questions: If I'm writing a web crawler named Goofy, what user agent ...
8
votes
2answers
4k views

Should I block bots from my site and why?

My logs are full of bot visitors, often from Eastern Europe and China. The bots are identified as Ahrefs, Seznam, LSSRocketCrawler, Yandex, Sogou and so on. Should I block these bots from my site and ...
2
votes
1answer
167 views

404 Hits from “PHPCrawl” on WP site

Forgive me if this is in the wrong area - I thought considering the topic, it belonged here (as opposed to in WP). Correct me or move it if I am wrong. I've been looking around one of the sites I ...
0
votes
1answer
53 views

Long user agent field in HTTP header

I have a HTTP sniffer application which runs on C++ , when I am getting the user agent field I get something like : Mozilla/5.0 (Windows NT 6.1; WOW655970812; fr=0ea2JbE8UfOtMxCAb....
2
votes
2answers
2k views

Correct usage of robots.txt file (“Googlebot-Image” and “Mediapartners-Google”)

This is the current robots.txt file I am using on a site of mine. I have excluded large parts of the IP.Board forum in order to try and cut down on duplicate content. I've also excluded some WordPress ...
5
votes
2answers
405 views

Plus sign in front of URLs in user agents

I run a small web crawler and had to decide on what user agent to use for it. Lists of crawler agents as well as Wikipedia suggest the following format: examplebot/1.2 (+http://www.example.com/bot....
6
votes
2answers
1k views

Why do some user agents have spam urls in them (and why are they always Opera/Presto User-Agents)?

If you go to (say) the last 100 entries (visits) to the botsvsbrowsers.com website (exact link, feel free to take a look: http://www.botsvsbrowsers.com/recent/listings/index.html ), you'd notice that ...
4
votes
3answers
2k views

Filtering desktop users by user agent string

In order to determine whether to serve the mobile or full version of my site by default, I use a service that examines requests to determine the type of device they originated from. However, calls to ...
8
votes
3answers
725 views

Tactics for dealing with misbehaving robots

I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site. Obviously we've had ...
2
votes
2answers
68 views

Question about duplicate groups (agents) in robots.txt

I'm looking at a corner case with robots.txt, and am curious if you have any guidance. Basically, I'm in a situation where multiple people will be making edits to the same robots.txt file. I've ...
-1
votes
2answers
319 views

How do I prevent duplicate indexing by Googlebot-Mobile?

I have only one version of my web-site, and it is optimised for both the desktop and the smartphone. I've noticed that Google crawls each page exactly twice: once with Googlebot/2.1, and again with ...
3
votes
1answer
674 views

Disallow all user agents except one using .htaccess?

I've been struggling to get this .htaccess working. The aim is to disallow all user agents besides my app. The app sends a GET request with a user agent of lets say 'AcmeUpdater'. Whenever I try to ...
1
vote
1answer
616 views

Block access to robots.txt file .htaccess

I read in this forum that this .htaccess code should block access to the robot.txt file: RewriteCond %{HTTP_USER_AGENT} ^Mozilla RewriteCond %{HTTP_USER_AGENT} !(Slurp|surfsafely) RewriteRule ^robots\...
3
votes
1answer
2k views

Preventing high bandwidth usage from Yandex

Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like ...
2
votes
1answer
604 views

Apache logging based on user agent

I'd like to update my apache logging so that its destination is based on the user agent string or file type. Specifically, I would like to send my googlebot requests to their own log file, and I ...
1
vote
1answer
796 views

Which browser contains “com.google.GooglePlus” in the agent string?

I noticed the following agent user strings hitting my website and wondered which browser is this user string for? Mozilla/5.0 (iPad2; iPad; U; CPU OS 5_1_1 like Mac OS X; en_GB) com.google.GooglePlus/...
0
votes
1answer
410 views

How to secure robots.txt file?

I would like for User-agents to index my relative pages only without accessing any directory on my server. As initial thought, i had this version in mind: User-agent: * Disallow: */* Sitemap: ...
2
votes
4answers
4k views

How should I deal with user agent parsing in logs?

My web app project includes logging functionality so we can see where visitors are coming from (referrer URL), what the popular user agents are, what pages are most popular, etc. The log is stored in ...
2
votes
1answer
255 views

Java/1.7.0_07 as user agent

almost every day something (it should be a bot) visits all of my pages on my website (amongst other things it doesn't correctly handle spaces in urls) with Java/1.7.0_07 (or different java versions) ...
4
votes
2answers
5k views

Why would an IE8 in a desktop has a 'Tablet PC 2.0' in its user-agent string?

I am just curious, why would a windows 7 desktop, installed with ie8, have Tablet PC 2.0 in its user agent string. Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0....
7
votes
2answers
803 views

How do you direct iPhone/Android browser to m.example.com?

I have some special coding that only works for HTML5 browsers (simple geolocation stuff). I want to redirect iPhone and Android users to the m.example.com version of my website. How is the best way ...
3
votes
2answers
2k views

What is a “SiteCon” user agent?

I've found some unusual activity in my site's Apache logs and every hit has this UserAgent string: "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-gb; SiteCon/8.8.14)" What is SiteCon?