Tag Info

Hot answers tagged

8

I'm the primary designer and author of a fairly large-scale web crawler (see http://metadatalabs.com/mlbot). What you're asking touches on a topic that's very important to us--perhaps the most important part of running a crawler: that of politeness. First: the reason for the "Mozilla" thing is to tell the site what your browser capabilities are. If your ...


8

I would see no reason to unblock Yahoo's search bot. There are 2 ways Yahoo could be using the Bing Engine. They could integrate it with their own and set a hybrid crawler loose on the net or they could just be sending the queries into the Bing Engine and spitting out the results they get back. From the document Joel linked to there is another document ...


8

Yes. Assuming that your agent names are specified correctly, it looks like this should work. Here is a resource if you want to read more. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt


7

Looking at bots vs. browsers, they display every user agent that's ever visited their page. Some clever spammer realised that this would be a clever way to drive traffic to their sites, because webmasters/anyone looking at the site is probably going to wonder why there's a url in the user agent, think it's a new specification or something, and visit the url ...


7

This looks to me like a Media Center PC with the tablet edition of Windows 7 or Windows 2008 Server R2 installed (Both Identify as NT 6.1). Hardware like this shows up with a similar string. Take a look at User Agent Strings for more help. Personally I avoid using browser strings to select for tablet and mobile, its much more practical to use screen ...


7

It appears that they've just started rolling out Bing search results on Yahoo! for about 25% of the searches, with the goal of completely switching over in August/September. I would bet that the Yahoo spider isn't even running any more and if it is, it's not very relevant.


7

Note: Not a direct answer, but IMO a valuable contribution. Of course this answer is dependent on your requirements, but I think many readers developing for consumers will find it useful and relevant, especially in the future. To such a fine degree, I don't. Of course, user-agent detection is great to serve an optimized mobile version for devices with ...


6

You can block bots but it depends on what you want for your website. You can block search engine bots if you don't want to see your website indexed in a particular search engine. Example: Yandex is russian search engine. You can block its bot if your business is not targeting Russia. You can block SEO bots if you don't want to use their web analytics ...


5

Mozilla/2.0 and Mozilla/5.0 are both references to the Mozilla browser. It has become largely meaningless, with many crawlers using it, but should tell the site to treat your crawler as it would any random user browsing with a regular browser. It is however good etiquette to include an URL linking to a page about who you are and why you are crawling in the ...


5

Whether or not crawlers honor your robots.txt is entirely an on-your-honor based system. Nothing you put in that file is going to prevent a "fake" crawler from doing anything. With regards to User-agent:, that value is completely voluntary as well. You can instruct your browser, or any other HTTP client to send whatever value you want for that header.


5

Here is a robots.txt file that will allow Google, Bing, and Yahoo to crawl the site while disallowing all other crawling: User-Agent: * Disallow: / User-Agent: googlebot Disallow: User-Agent: bingbot Disallow: User-agent: slurp Disallow: Some crawlers ignore robots.txt entirely and crawl whatever they feel like. Some crawlers impersonate Googlebot or ...


4

As paulmorriss mentioned in the comments, most major web servers do this automatically anyways, with their server logs. These logs contain UA strings, IP addresses, and much more. I don't know of any law on the books that would explicitly forbid this, but I would say that you should probably include in the Terms of Service the fact that you collect this ...


4

Finally realised what the problem was when, after deleting the tag in both files, the error "Invalid command '\xef\xbb\xbf EF BB BF = BOM Moral of the story: Always check your encoding* and hex dump of your files when you get weird issues. *On Notepad++ you can use the Encoding -> Encode in UTF-8 without BOM option to remove it


4

That's going to block your entire website from being crawled. No There is no such thing as securing your robots.txt. If you don't want to keep visitors out of your directory root you need to prevent that using more secure means. Putting a blank index.html file will easily do the trick. If you're running Apache you can also do it easily using htaccess.


4

I raised a search enquiry with Yahoo on Monday - the acknowledgement included the following note on the transition to Bing which you might find relevant: Yahoo! Search is excited to reach a new stage in the Yahoo! and Microsoft Search Alliance; the integration of Microsoft algorithmic results is near complete for the U.S. and Canada marketplace. ...


4

OMG, are you really keeping your own database up to date? I'm so sorry about you... First tip: If you only need a very simple and minimal solution to detect brand and model, go with WURFL if you want it for free or DeviceAtlas if you can afford it. The second one works probably better (just my personal opinion). Also, take a look at this comparision ...


4

While attempting to block bots can help in free up resources and clean up your logs it’s important to note that robots.txt and even using the meta tag on pages noindex does not actually stop bots visiting your site. They can still crawl your site occasionally to see if the denied from robots has been removed. A lot of bots don’t even use a user agent and ...


3

You failed to share the ip address of the request - so its a bit more difficult to answer the who - but a good idea of why could be a simple scan to check and see if the website is working with all browsers. This is standard for many developers when they make a change - services like www.BrowserShots.org generally show up this way.


3

Browser manufacturers like to cram unecessary information into their user-agent strings for a variety of reasons. Originally it was to fool browser sniffers that were excluding IE from their website (that's why you see Mozilla/4.0 (compatible; in IE's user-agent). It looks like Microsoft is cramming everything IE8 can be used with into its user-agent which ...


3

Completely agree that using a SQL query is not the most effective approach to this. However, if the logs you happen to need are in SQL already, and it's a one off or data sizes a not large enough to warrant improving the approach, here's a handy expression that can be used as a starting point... Based on Sarath's post here: ...


3

Those are from the Google+ app running on the iPad and iPhone. It's running a modified Chrome browser. The mention of Mozilla/5.0 has to do with the history of User Agent Strings, and nothing to do with who built the software. See more here: http://webaim.org/blog/user-agent-string-history/


3

You are already telling them because you have one site using a single set of URLs. They visit twice because each bot evaluates differently. In the end it goes in the same index but the metadata is different which lets them serve different results to standard and mobile queries.


3

Does the position of things in the file matter? No. The groups can be in any order. I'm now a little confused after reading that the addition of User-agent: Googlebot may override certain other values. Any one crawler will only look at one group in the robots.txt file. The crawler will only look at the most specific group, determined by the length ...


2

Firefox 3.x and many others do. Can you post the full agent string that you're having problems with? As it sounds like it could be a custom script that is generating the request and pretending to be a certain agent. But is incorrectly creating the ident string. Or your check against the agent string is incorrect. Edit: The strings you posted are being ...


2

Using the programming functions in SQL is a pretty bad way to deal with this. A decent method would probably be to add an extra field to the table for the browser name (or ID from another table), and every time you store a row in the database, use your programming language there (e.g. PHP) to decide on the user agent and store that. Then you can do a select ...


2

As Gisle Hannemyr mentioned in a comment, the best way to do this is to require logins of all users, and do not provide the restricted content to anyone who isn't logged in. If you can't require logins for some reason, there are still a couple of fallbacks you can use (disclaimer: both of them are either partly or completely my fault): The OWASP ...


2

You can piggyback on work other people have done in identifying bad IPs by using an Apache module which interfaces with Project Honeypot's IP blacklist. If you're doing this on a large scale, it would probably be polite to offer to run a honeypot.


2

Major search engines do this too. After a period (between 6 and 18 months), logs have all the IPs replaced by another unique identifier generated randomly for each IP. That way you can measure user behavior without identifying the user. You can do this immediately instead by converting IPs into something else using a hash and tagging with Geo-Location ...


2

Why not an Internet-connected Refrigerator Browser? Whirlpool make something like that in 2k. See also: Browser Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Whirlpool BD2) Browser Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Whirlpool BD2; FunWebProducts) Browser Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; Whirlpool BD2) ...


2

Use a robots.txt crawl delay as described in http://help.yandex.com/search/?id=1112639 Example: User-agent: Yandex Crawl-delay: 2 # specifies a 2 second timeout Before you start banning this bot, you should first verify that your logs are actually Yandex and not someone else who is spoofing the user agent to look like they are yandex. A tactic used by ...



Only top voted, non community-wiki answers of a minimum length are eligible