Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [user-agent]

An http header field that identifies the browser and operating system.

0
votes
0answers
11 views

Why would a bot net keep loading my site?

In blocking a lot of scrapping activity, I found a puzzling situation - that my site is accessed about 35,000 times per day by around 30,000 different IPs. I started blocking these IPs with a script, ...
2
votes
0answers
18 views

Assessing TLS capabilities by examining user agent strings from log files

I have a long list of browser user agent strings pulled from website access logs which I would like to analyze to determine the TLS capabilities of our site's visitors in order to get a ballpark ...
4
votes
2answers
31 views

Would modifying my website when used by our app with a custom user agent affect SEO?

Here is the deal, I have an app that gets content via a webview of my website on the internet. The only thing am changing is that am setting a custom user-agent for this webview app, so I am able to ...
1
vote
0answers
313 views

What is user agent ;__CT_JOB_ID__:?

I've been searching the web for an explanation of the sudden influx we've gotten of user agents of this format: ;__CT_JOB_ID__:68a72eee-353c-4b0b-82ae-0cb6c8ca3e6e; Each visitor has a unique IP. So ...
2
votes
0answers
49 views

User agent of BingWeb - bot or mobile agent?

When I scan my logs for user agents containing Bing (capitalized), I find two types of user agents: those containing BingPreview and those containing BingWeb. Bing lists BingPreview as one of their ...
0
votes
1answer
78 views

Why would a user agent string contain both “Samsung” and “AppleWebKit”?

Investigating an issue on a website, I see an entry in the Apache log with this User Agent setting: Mozilla/5.0 (Linux; Android 8.0.0; SAMSUNG SM-G955U Build/R16NW) AppleWebKit/537.36 (KHTML, like ...
2
votes
0answers
516 views

Is it possible to override the Browser name in Google Analytics?

I was curious how many of my visitors were using the Tor browser and wrote a script to detect it. Since that browser sets the user agent as Windows/Firefox, it means the data (however small) in Google ...
1
vote
1answer
1k views

Is “Mozilla/4.0 (compatible;)” a valid user-agent?

I've been getting an increasing number of requests (currently at around 100K per day) with user agent Mozilla/4.0 (compatible;). Usually, when I see a user-agent or requests that look suspicious I do ...
5
votes
1answer
384 views

Are user agent names case-sensitive in robots.txt?

I'm blocking various bots in robots.txt and I was wondering if their names are case-sensitive. For example: User-agent: grapeshot Disallow: / Would it matter if I use grapeshot or GrapeShot?
3
votes
1answer
367 views

What is the “google-xrawler” user agent used for?

I'm getting over 100 requests per day from user agent google-xrawler. The requests are all for the same page. Some of the IP addresses are known to be Google IPs, although they don't have valid ...
3
votes
2answers
76 views

Could changing the download link and screenshots based on user agent be considered cloaking?

I sell a software product. I currently detect if the user agent identifies itself as Mac somehow. if so, the server generates the landing page, so its default download link is for Mac and shows a ...
10
votes
3answers
883 views

Was user-agent identification used for some scripting attack techique?

Apache access log entries on my site are typically like this one: 207.46.13.174 - - [31/Oct/2016:10:18:55 +0100] "GET /contact HTTP/1.1" 200 256 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www....
5
votes
1answer
1k views

Why do many websites block requests from common HTTP libraries by user-agent?

Writing spiders, I have noticed that many sites will return a 403 error if I hit them from popular HTTP software libraries, unless I manually override the default User-Agent header used by the library....
0
votes
2answers
480 views

Googlebot-Mobile ignoring robots.txt, pretending to be Googlebot

I'm very disappointed that Googlebot is apparently ignoring my robots.txt. I have the following sole entry for my /robots.txt: location = /robots.txt { return 200 "## $host ## \n #Dear ...
4
votes
1answer
176 views

User Agents with Hash

104.245.34.141 - - [09/Apr/2016:11:51:35 -0400] "GET /22327 HTTP/1.1" 200 3516 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0#19d5fd" 86.105.50.203 - - [09/Apr/2016:...
8
votes
4answers
629 views

User-Agent with base64-encoded component?

(Bounty question at the bottom) I'm having an issue with a client accessing our site, and the root cause is that the WAF (Web Application Firewall) doesn't like their User-Agent string: User-Agent: ...
4
votes
3answers
3k views

How to determine if a user agent string has proper syntax or might be a hacking attempt?

I was checking my server via awstats to see who was visiting my site and I have a user-agent of the following value: }__test|O:21:\ Some research has lead me to believe that someone was trying to ...
2
votes
1answer
40 views

Applications that meet the definition of a User Agent under the W3C

I'm reading through the W3C's Definition of User Agent: Definition: A user agent is any software that retrieves, renders and facilitates end user interaction with Web content, or whose user ...
0
votes
1answer
37 views

Can't get my browser to show me the mobile version of my website

I'm trying to debug something in the mobile version of my website, but I just can't make it show up on my desktop browser. I'm using the plugin "User Agent Switcher" on Firefox with this user agent: ...
2
votes
1answer
109 views

Yandex mobile services robot - is it really a mobile user agent?

As you can tell, I want my website indexed properly in as many search engines as possible. I'm looking at Yandex and using its "Server Response Check" tool which is similar to "Fetch as google" tool ...
1
vote
1answer
41 views

Handling visitors computers that don't report languages or countries

I added google analytics to the mobile version of my site and three days later it seems that most of the visitors either do not have a language set or a country code set. Should I assume every ...
2
votes
3answers
2k views

How to block a user-agent that has a spacing in its name?

I got a hit from a crawler with a user-agent called DV CRAWLER which is an abvious a spam-bot. I tried to block it in both .htacess and nginx configuration as I'm running nginx as a reverse proxy in ...
0
votes
1answer
277 views

How to find out which links of website posted on Twitter

I recenlty noticed many requests from Twitterbot user agent in my website's logs which I believe it doesn't crawl by itself unless someone is posting links of my website on Twitter. These posts might ...
3
votes
1answer
574 views

In Webtrends, what browser is “Mozilla <Version Unknown>”?

I am getting a ton of visits from a browser that is showing up in my Webtrends stats as "Mozilla" with version listed as "Version Unknown." It's almost 14% of my page views. This is separate from ...
2
votes
2answers
537 views

Is it okay to block all user agents that start with “PHP/5.”?

I'm getting a lot of spambots from China (and elsewhere, likely botnets) accessing my blog which have a user agent string starting with PHP/5.. These include such nonsense values as PHP/5.3.70 and PHP/...
1
vote
1answer
4k views

Detecting Android tablets vs phones with User Agent

I inherited a site that has PHP code that decides to either serve the dedicated mobile site or desktop site to the user. I would prefer that tablets get served the mobile site. Currently Android ...
3
votes
2answers
5k views

What browser is the “Apache-HttpClient/UNAVAILABLE (java 1.4)” user agent?

Some of my users are logging in with this browser: Apache-HttpClient/UNAVAILABLE (java 1.4). What browser is this?
4
votes
1answer
814 views

Can I block requests that have no user agent because they cause 404 errors?

Is it OK to block requests that contain no user agent? I am getting random 404 errors on URLs I have never used. These requests have no user agent. If this is the case, will it affect any of my SEO?...
0
votes
1answer
603 views

What percent of all active IP + User Agent combinations are not unique?

It's fairly common practice to define a unique user by a 1-to-1 hash of IP + User Agent. But not all unique users have a unique IP + User Agent combination. Thus: collisions - events misattributed to ...
10
votes
1answer
922 views

Combine user-agents in robots.txt

Can user-agents be listed together, followed by their common rules like so in a robots.txt? User-agent: Googlebot User-agent: ia_archiver #Alexa User-agent: BingPreview #Microsoft User-agent: bingbot ...
2
votes
2answers
2k views

How to block user agent from accessing my WordPress site with htaccess?

I have noticed large amount of traffic coming to my site from different IP's with user agent. Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/) I don't have much experience ...
1
vote
1answer
1k views

Identify this user agent - claims to be IE7 on Windows 7

We have a tracking pixel in one of our emails and our the most frequent user agent is this strange one: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0....
2
votes
2answers
2k views

Finding different Windows XP Service Packs via User-Agent string

I am planning to get rid of the SHA1 certificate signing algorithm in our certificates and switch over to SHA-256. Problem is, that Windows XP (before SP3) is not supporting this. Therefore I need to ...
1
vote
2answers
74 views

How can we find real source of suspected user entries?

I have a website where I keep logging the user entries. I found some anonymous entries in my table and I suspect they are using some kind of proxy switchers. How can I find the real source of these ...
2
votes
3answers
2k views

Can web crawlers that visit my site fake their user agent?

I want to write a robots.txt file for my website and allow the famous bots (Google, Bing, and Yahoo) to crawl my website, but deny the rest. I want to know if I add User-agent: Googlebot, will fake ...
2
votes
1answer
209 views

404 Hits from “PHPCrawl” on WP site

Forgive me if this is in the wrong area - I thought considering the topic, it belonged here (as opposed to in WP). Correct me or move it if I am wrong. I've been looking around one of the sites I ...
0
votes
1answer
87 views

Long user agent field in HTTP header

I have a HTTP sniffer application which runs on C++ , when I am getting the user agent field I get something like : Mozilla/5.0 (Windows NT 6.1; WOW655970812; fr=0ea2JbE8UfOtMxCAb....
2
votes
2answers
4k views

Correct usage of robots.txt file (“Googlebot-Image” and “Mediapartners-Google”)

This is the current robots.txt file I am using on a site of mine. I have excluded large parts of the IP.Board forum in order to try and cut down on duplicate content. I've also excluded some WordPress ...
8
votes
2answers
847 views

Plus sign in front of URLs in user agents

I run a small web crawler and had to decide on what user agent to use for it. Lists of crawler agents as well as Wikipedia suggest the following format: examplebot/1.2 (+http://www.example.com/bot....
2
votes
2answers
129 views

Question about duplicate groups (agents) in robots.txt

I'm looking at a corner case with robots.txt, and am curious if you have any guidance. Basically, I'm in a situation where multiple people will be making edits to the same robots.txt file. I've ...
-1
votes
2answers
368 views

How do I prevent duplicate indexing by Googlebot-Mobile?

I have only one version of my web-site, and it is optimised for both the desktop and the smartphone. I've noticed that Google crawls each page exactly twice: once with Googlebot/2.1, and again with ...
4
votes
3answers
3k views

Filtering desktop users by user agent string

In order to determine whether to serve the mobile or full version of my site by default, I use a service that examines requests to determine the type of device they originated from. However, calls to ...
11
votes
2answers
4k views

Should I block bots from my site and why?

My logs are full of bot visitors, often from Eastern Europe and China. The bots are identified as Ahrefs, Seznam, LSSRocketCrawler, Yandex, Sogou and so on. Should I block these bots from my site and ...
1
vote
1answer
811 views

Which browser contains “com.google.GooglePlus” in the agent string?

I noticed the following agent user strings hitting my website and wondered which browser is this user string for? Mozilla/5.0 (iPad2; iPad; U; CPU OS 5_1_1 like Mac OS X; en_GB) com.google.GooglePlus/...
2
votes
1answer
402 views

Java/1.7.0_07 as user agent

almost every day something (it should be a bot) visits all of my pages on my website (amongst other things it doesn't correctly handle spaces in urls) with Java/1.7.0_07 (or different java versions) ...
8
votes
3answers
10k views

Interpretation of empty User-agent

How should I interpret a empty User-agent? I have some custom analytics code and that code has to analyze only human traffic. I have got a working list of User-agents denoting human traffic, and bot ...
5
votes
4answers
3k views

What does it mean when a User-Agent has another User-Agent inside it?

Basically, sometimes the user-agent will have its normal user-agent displayed, then at the end it will have the "User-Agent: " tag displayed, and right after it another user-agent is shown. Sometimes, ...
6
votes
2answers
2k views

Why do some user agents have spam urls in them (and why are they always Opera/Presto User-Agents)?

If you go to (say) the last 100 entries (visits) to the botsvsbrowsers.com website (exact link, feel free to take a look: http://www.botsvsbrowsers.com/recent/listings/index.html ), you'd notice that ...
3
votes
1answer
847 views

Disallow all user agents except one using .htaccess?

I've been struggling to get this .htaccess working. The aim is to disallow all user agents besides my app. The app sends a GET request with a user agent of lets say 'AcmeUpdater'. Whenever I try to ...
0
votes
1answer
972 views

How to secure robots.txt file?

I would like for User-agents to index my relative pages only without accessing any directory on my server. As initial thought, i had this version in mind: User-agent: * Disallow: */* Sitemap: ...