An http header field that identifies the browser and operating system.
20
votes
4answers
752 views
Is Yahoo Search the same as Bing Search now?
We had some serious issues with Yahoo's (incredibly) badly written spider in the past, and as a result we blocked them. Marco Arment of Tumblr also shared his frustrations with us on 8/31/2009, which ...
8
votes
3answers
291 views
Tactics for dealing with misbehaving robots
I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site.
Obviously we've had ...
7
votes
2answers
535 views
How do you direct iPhone/Android browser to m.example.com?
I have some special coding that only works for HTML5 browsers (simple geolocation stuff). I want to redirect iPhone and Android users to the m.example.com version of my website. How is the best way ...
5
votes
2answers
405 views
Should I block bots from my site and why?
My logs are full of bot visitors, often from Eastern Europe and China.
The bots are identified as Ahrefs, Seznam, LSSRocketCrawler, Yandex, Sogou and so on.
Should I block these bots from my site and ...
5
votes
2answers
323 views
Why do some user agents have spam urls in them (and why are they always Opera/Presto User-Agents)?
If you go to (say) the last 100 entries (visits) to the botsvsbrowsers.com website (exact link, feel free to take a look: http://www.botsvsbrowsers.com/recent/listings/index.html ), you'd notice that ...
5
votes
3answers
291 views
What does it mean when a User-Agent has another User-Agent inside it?
Basically, sometimes the user-agent will have its normal user-agent displayed, then at the end it will have teh "User-Agent: " tag displayed, and right after it another user-agent is shown. Sometimes, ...
4
votes
3answers
4k views
Mobile phone detection (brand, model, browser etc)
What do you use to detect visitor's mobile phone, down to the model if possible?
Currently we maintain our own database but it's really getting behind due to lack of manpower to maintain it, so we ...
4
votes
2answers
329 views
Is collecting IP address and user agents on form submission illegal in the U.S.?
I'm creating a website where I collect the name, email, state, and an optional photo of a user. Before the user submits the form, they have to agree to the terms of the site. For testing and ...
4
votes
3answers
161 views
Filtering desktop users by user agent string
In order to determine whether to serve the mobile or full version of my site by default, I use a service that examines requests to determine the type of device they originated from. However, calls to ...
3
votes
2answers
2k views
Why would an IE8 in a desktop has a 'Tablet PC 2.0' in its user-agent string?
I am just curious, why would a windows 7 desktop, installed with ie8, have Tablet PC 2.0 in its user agent string.
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2;
.NET CLR ...
3
votes
1answer
328 views
Disallow all user agents except one using .htaccess?
I've been struggling to get this .htaccess working. The aim is to disallow all user agents besides my app. The app sends a GET request with a user agent of lets say 'AcmeUpdater'.
Whenever I try to ...
3
votes
1answer
526 views
Preventing high bandwidth usage from Yandex
Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like ...
3
votes
2answers
173 views
Does the user agent in any regular browser contain 'bot' or 'crawl'?
Does the user agent in any regular browser contain 'bot' or 'crawl'?
I check the user agent on my site to see if it is coming from a bot or not. If it is, I can do some little optimizations since ...
2
votes
3answers
741 views
What user agent should I set?
There is Ask bot, which sets this header:
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
Considering this, I have the following questions:
If I'm writing a web crawler named Goofy, what user agent ...
2
votes
4answers
553 views
How should I deal with user agent parsing in logs?
My web app project includes logging functionality so we can see where visitors are coming from (referrer URL), what the popular user agents are, what pages are most popular, etc. The log is stored in ...
2
votes
1answer
310 views
Apache logging based on user agent
I'd like to update my apache logging so that its destination is based on the user agent string or file type.
Specifically, I would like to send my googlebot requests to their own log file, and I ...
2
votes
1answer
151 views
A single request appears to have come from all the browsers? Should I be worried?
I was looking over my site access logs when I noticed a request with the following user agent string:
"Mozilla/5.0 (Windows; U; Windows NT
5.1; en-US; rv:1.9.2.12) Gecko/20101026
...
2
votes
2answers
241 views
What is a “SiteCon” user agent?
I've found some unusual activity in my site's Apache logs and every hit has this UserAgent string:
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-gb; SiteCon/8.8.14)"
What is SiteCon?
2
votes
2answers
26 views
Question about duplicate groups (agents) in robots.txt
I'm looking at a corner case with robots.txt, and am curious if you have any guidance.
Basically, I'm in a situation where multiple people will be making edits to the same robots.txt file.
I've ...
1
vote
1answer
267 views
Which browser contains “com.google.GooglePlus” in the agent string?
I noticed the following agent user strings hitting my website and wondered which browser is this user string for?
Mozilla/5.0 (iPad2; iPad; U; CPU OS 5_1_1 like Mac OS X; en_GB) ...
1
vote
1answer
109 views
Java/1.7.0_07 as user agent
almost every day something (it should be a bot) visits all of my pages on my website (amongst other things it doesn't correctly handle spaces in urls) with Java/1.7.0_07 (or different java versions) ...
1
vote
3answers
329 views
Interpretation of empty User-agent
How should I interpret a empty User-agent? I have some custom analytics code and that code has to analyze only human traffic. I have got a working list of User-agents denoting human traffic, and bot ...
1
vote
1answer
90 views
In User Agent strings, what does “AskTbTRL” denote?
Here are a bunch of examples:
http://www.botsvsbrowsers.com/listings.asp?search=asktbtrl
I would've guessed an Ask.com-related bot, but apparently not according to their FAQ.
1
vote
1answer
85 views
what is user agent Whirlpool BD2; H010818
what is this user agent
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Whirlpool BD2; H010818)
this came from a ip address tracing to the organisation Websense
the question also here
thanks
1
vote
2answers
164 views
What browser is sending user agent beginning mozilla/5.0+, tramslates & into &
We've got a website which has been running for a few years now. One of our customers has just started having an intermittent problem. Looking at our iis6.0 logs the service works correctly when they ...
1
vote
1answer
297 views
Block access to robots.txt file .htaccess
I read in this forum that this .htaccess code should block access to the robot.txt file:
RewriteCond %{HTTP_USER_AGENT} ^Mozilla
RewriteCond %{HTTP_USER_AGENT} !(Slurp|surfsafely)
RewriteRule ...
1
vote
1answer
93 views
What is user agent Issbot
I am seeing a lot of traffic to my website from user agent Issbot. This certainly looks like a bot, but is not listed in the www.user-agents.org listing, and I have not been able to find anything ...
0
votes
1answer
246 views
How to secure robots.txt file?
I would like for User-agents to index my relative pages only without accessing any directory on my server.
As initial thought, i had this version in mind:
User-agent: *
Disallow: */*
Sitemap: ...
0
votes
2answers
271 views
SSL Certificate is Untrusted how to fix it
How can i fix the untrusted ? And why it appears ? What may cause this problem.This is in FF but in Chrome is red https with x marked on it.
0
votes
1answer
782 views
BingPreview user agent
I'm getting several invalid requests from a "BingPreview" user agent. The complete user agent header is: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
The ...
0
votes
1answer
77 views
What browser or bot has the user agent of “SD-1.30.1”?
I have tried checking on Google but to no avail. Does anyone recognize this user agent?
0
votes
2answers
148 views
Untrusted user-agent?
My logs recently showed the following user-agent:
samsung-gt-s5620 UNTRUSTED/1.0
Why would a user-agent have untrusted in caps in its name?
0
votes
1answer
299 views
Website won't load in Firefox 3.6.13
After the latest version of Firefox came out, I've been having problems with my website. The site is: http://mojheroj.com It works fine in all other browsers, as well as older versions of Firefox and ...
0
votes
0answers
181 views
List of most common bot user-agents? [closed]
I'm trying to find a list of the most common bot user-agents (say 10 or so) but I can't find much of anything; all I'm finding is lists that contains thousands of user-agents.
All I need is the most ...
-2
votes
2answers
89 views
How do I prevent duplicate indexing by Googlebot-Mobile?
I have only one version of my web-site, and it is optimised for both the desktop and the smartphone.
I've noticed that Google crawls each page exactly twice: once with Googlebot/2.1, and again with ...
