Hot answers tagged user-agent
8
I would see no reason to unblock Yahoo's search bot. There are 2 ways Yahoo could be using the Bing Engine. They could integrate it with their own and set a hybrid crawler loose on the net or they could just be sending the queries into the Bing Engine and spitting out the results they get back.
From the document Joel linked to there is another document ...
7
Looking at bots vs. browsers, they display every user agent that's ever visited their page. Some clever spammer realised that this would be a clever way to drive traffic to their sites, because webmasters/anyone looking at the site is probably going to wonder why there's a url in the user agent, think it's a new specification or something, and visit the url ...
7
This looks to me like a Media Center PC with the tablet edition of Windows 7 or Windows 2008 Server R2 installed (Both Identify as NT 6.1). Hardware like this shows up with a similar string.
Take a look at User Agent Strings for more help.
Personally I avoid using browser strings to select for tablet and mobile, its much more practical to use screen ...
7
It appears that they've just started rolling out Bing search results on Yahoo! for about 25% of the searches, with the goal of completely switching over in August/September. I would bet that the Yahoo spider isn't even running any more and if it is, it's not very relevant.
6
Note: Not a direct answer, but IMO a valuable contribution. Of course this answer is dependent on your requirements, but I think many readers developing for consumers will find it useful and relevant, especially in the future.
To such a fine degree, I don't.
Of course, user-agent detection is great to serve an optimized mobile version for devices with ...
6
You can block bots but it depends on what you want for your web site.
You can block search engine bots if you don't want to see your web site indexed in particular search engine.
Example: Yandex is russian search engine. You can block it if your business is not targeting Russia.
You can block SEO bots if you don't want to use web analytics of it.
Example: ...
4
I raised a search enquiry with Yahoo on Monday - the acknowledgement included the following note on the transition to Bing which you might find relevant:
Yahoo! Search is excited to reach a new stage in the Yahoo! and
Microsoft Search Alliance; the integration of Microsoft algorithmic
results is near complete for the U.S. and Canada marketplace. ...
4
I'm the primary designer and author of a fairly large-scale web crawler (see http://metadatalabs.com/mlbot). What you're asking touches on a topic that's very important to us--perhaps the most important part of running a crawler: that of politeness.
First: the reason for the "Mozilla" thing is to tell the site what your browser capabilities are. If your ...
4
Mozilla/2.0 and Mozilla/5.0 are both references to the Mozilla browser. It has become largely meaningless, with many crawlers using it, but should tell the site to treat your crawler as it would any random user browsing with a regular browser.
It is however good etiquette to include an URL linking to a page about who you are and why you are crawling in the ...
4
As paulmorriss mentioned in the comments, most major web servers do this automatically anyways, with their server logs. These logs contain UA strings, IP addresses, and much more.
I don't know of any law on the books that would explicitly forbid this, but I would say that you should probably include in the Terms of Service the fact that you collect this ...
4
Finally realised what the problem was when, after deleting the tag in both files, the error "Invalid command '\xef\xbb\xbf
EF BB BF = BOM
Moral of the story: Always check your encoding* and hex dump of your files when you get weird issues.
*On Notepad++ you can use the Encoding -> Encode in UTF-8 without BOM option to remove it
4
That's going to block your entire website from being crawled.
No
There is no such thing as securing your robots.txt. If you don't want to keep visitors out of your directory root you need to prevent that using more secure means. Putting a blank index.html file will easily do the trick. If you're running Apache you can also do it easily using htaccess.
3
Those are from the Google+ app running on the iPad and iPhone. It's running a modified Chrome browser.
The mention of Mozilla/5.0 has to do with the history of User Agent Strings, and nothing to do with who built the software. See more here: http://webaim.org/blog/user-agent-string-history/
3
You failed to share the ip address of the request - so its a bit more difficult to answer the who - but a good idea of why could be a simple scan to check and see if the website is working with all browsers.
This is standard for many developers when they make a change -
services like www.BrowserShots.org generally show up this way.
3
Browser manufacturers like to cram unecessary information into their user-agent strings for a variety of reasons. Originally it was to fool browser sniffers that were excluding IE from their website (that's why you see Mozilla/4.0 (compatible; in IE's user-agent). It looks like Microsoft is cramming everything IE8 can be used with into its user-agent which ...
3
OMG, are you really keeping your own database up to date? I'm so sorry about you...
First tip: If you only need a very simple and minimal solution to detect brand and model, go with WURFL if you want it for free or DeviceAtlas if you can afford it. The second one works probably better (just my personal opinion).
Also, take a look at this comparision ...
3
While attempting to block bots can help in free up resources and clean up your logs it’s important to note that Robots.txt and even using the meta tag on pages noindex does not actually stop bots visiting your site. They can still crawl your site occasionally to see if the denied from robots has been removed. A lot of bots don’t even use a user agent and ...
2
Firefox 3.x and many others do.
Can you post the full agent string that you're having problems with? As it sounds like it could be a custom script that is generating the request and pretending to be a certain agent. But is incorrectly creating the ident string. Or your check against the agent string is incorrect.
Edit:
The strings you posted are being ...
2
Why not an Internet-connected Refrigerator Browser?
Whirlpool make something like that in 2k.
See also:
Browser Mozilla/4.0 (compatible; MSIE
5.5; Windows 98; Whirlpool BD2)
Browser Mozilla/4.0 (compatible; MSIE
5.5; Windows 98; Whirlpool BD2; FunWebProducts)
Browser Mozilla/4.0 (compatible; MSIE
5.5; Windows NT 5.0; Whirlpool BD2)
...
2
Completely agree that using a SQL query is not the most effective approach to this. However, if the logs you happen to need are in SQL already, and it's a one off or data sizes a not large enough to warrant improving the approach, here's a handy expression that can be used as a starting point...
Based on Sarath's post here: ...
2
Using the programming functions in SQL is a pretty bad way to deal with this. A decent method would probably be to add an extra field to the table for the browser name (or ID from another table), and every time you store a row in the database, use your programming language there (e.g. PHP) to decide on the user agent and store that. Then you can do a select ...
2
Major search engines do this too.
After a period (between 6 and 18 months), logs have all the IPs replaced by another unique identifier generated randomly for each IP. That way you can measure user behavior without identifying the user.
You can do this immediately instead by converting IPs into something else using a hash and tagging with Geo-Location ...
2
Use a robots.txt crawl delay as described in http://help.yandex.com/search/?id=1112639
Example:
User-agent: Yandex
Crawl-delay: 2 # specifies a 2 second timeout
Before you start banning this bot, you should first verify that your logs are actually Yandex and not someone else who is spoofing the user agent to look like they are yandex. A tactic used by ...
2
It seems to be Java crawler. Anyone could have written it or used existing one. I would say you don't need to worry as www is kind of "read only" so no crawl will harm your page. However this is possible way for stealing content or analyse it for an undesired purpose.
It depends on you, will you block it :)
2
You are already telling them because you have one site using a single set of URLs.
They visit twice because each bot evaluates differently. In the end it goes in the same index but the metadata is different which lets them serve different results to standard and mobile queries.
1
If you want to analyze only "human traffic" I would not count the ones with empty or missing user agent string. In my experience almost any browser will always send one. Even most privacy plugins or extensions rather fake (include other OS or Client name) or "normalize" (e.g. no release numbers) or randomize (e.g. sometimes FF, sometimes IE strings) the UA ...
1
Almost certainly this is a bug in the software that is accessing your web site - a browser, a browser plugin, or a script/command-line program like wget. It's like someone tried to modify the User-agent header but instead pasted it inside a standard header.
Unless the user's particular IP is causing problems overloading the server, there is no real action ...
1
I think that by including a complete HTML anchor in the user agent the spammer is simply hoping that the target website is going to display the complete user agent unencoded (so the HTML is rendered), possibly in unprotected stats pages, and thus benefit from some free linkage.
This is a similar principal to referer spam.
Displaying an unencoded user agent ...
1
As Gisle Hannemyr mentioned in a comment, the best way to do this is to require logins of all users, and do not provide the restricted content to anyone who isn't logged in.
If you can't require logins for some reason, there are still a couple of fallbacks you can use (disclaimer: both of them are either partly or completely my fault):
The OWASP ...
1
You can piggyback on work other people have done in identifying bad IPs by using an Apache module which interfaces with Project Honeypot's IP blacklist. If you're doing this on a large scale, it would probably be polite to offer to run a honeypot.
Only top voted, non community-wiki answers of a minimum length are eligible
