Tag Info

Hot answers tagged

11

I'm the primary designer and author of a fairly large-scale web crawler (see http://metadatalabs.com/mlbot). What you're asking touches on a topic that's very important to us--perhaps the most important part of running a crawler: that of politeness. First: the reason for the "Mozilla" thing is to tell the site what your browser capabilities are. If your ...


10

I would see no reason to unblock Yahoo's search bot. There are 2 ways Yahoo could be using the Bing Engine. They could integrate it with their own and set a hybrid crawler loose on the net or they could just be sending the queries into the Bing Engine and spitting out the results they get back. From the document Joel linked to there is another document ...


10

Blocking No User Agent Blocking based on no user agent is a silly idea... a lot of users who like to remain anonymous through VPNS will often disable user agent and anything else that can be used to harvest data... And anonymity is growing. Also if the idea behind this is to save on resources it should be noted that most bots that are not legit search ...


7

You can link to a hidden page that, when visited, captures the useragent and IP address of the bot and then appends one or both of them to a .htaccess file which blocks them permanently. It's automated so you don't have to do anything to maintain it.


7

Mozilla/2.0 and Mozilla/5.0 are both references to the Mozilla browser. It has become largely meaningless, with many crawlers using it, but should tell the site to treat your crawler as it would any random user browsing with a regular browser. It is however good etiquette to include an URL linking to a page about who you are and why you are crawling in the ...


7

It appears that they've just started rolling out Bing search results on Yahoo! for about 25% of the searches, with the goal of completely switching over in August/September. I would bet that the Yahoo spider isn't even running any more and if it is, it's not very relevant.


7

Note: Not a direct answer, but IMO a valuable contribution. Of course this answer is dependent on your requirements, but I think many readers developing for consumers will find it useful and relevant, especially in the future. To such a fine degree, I don't. Of course, user-agent detection is great to serve an optimized mobile version for devices with ...


7

This looks to me like a Media Center PC with the tablet edition of Windows 7 or Windows 2008 Server R2 installed (Both Identify as NT 6.1). Hardware like this shows up with a similar string. Take a look at User Agent Strings for more help. Personally I avoid using browser strings to select for tablet and mobile, its much more practical to use screen ...


7

Looking at bots vs. browsers, they display every user agent that's ever visited their page. Some clever spammer realised that this would be a clever way to drive traffic to their sites, because webmasters/anyone looking at the site is probably going to wonder why there's a url in the user agent, think it's a new specification or something, and visit the url ...


7

Yes. Assuming that your agent names are specified correctly, it looks like this should work. Here is a resource if you want to read more. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt


6

According to the list at http://www.useragentstring.com/pages/Browserlist/ with over 9000 user agent strings from various browsers: 0 user agent strings of browsers contains the word "bot" 2 user agent strings of browsers contains the word "crawl" 0 user agent strings of browsers contains the word "spider" (The 2 which contains "crawl" is the following: ...


6

You can block bots but it depends on what you want for your website. You can block search engine bots if you don't want to see your website indexed in a particular search engine. Example: Yandex is russian search engine. You can block its bot if your business is not targeting Russia. You can block SEO bots if you don't want to use their web analytics ...


5

As paulmorriss mentioned in the comments, most major web servers do this automatically anyways, with their server logs. These logs contain UA strings, IP addresses, and much more. I don't know of any law on the books that would explicitly forbid this, but I would say that you should probably include in the Terms of Service the fact that you collect this ...


5

Whether or not crawlers honor your robots.txt is entirely an on-your-honor based system. Nothing you put in that file is going to prevent a "fake" crawler from doing anything. With regards to User-agent:, that value is completely voluntary as well. You can instruct your browser, or any other HTTP client to send whatever value you want for that header.


5

Here is a robots.txt file that will allow Google, Bing, and Yahoo to crawl the site while disallowing all other crawling: User-Agent: * Disallow: / User-Agent: googlebot Disallow: User-Agent: bingbot Disallow: User-agent: slurp Disallow: Some crawlers ignore robots.txt entirely and crawl whatever they feel like. Some crawlers impersonate Googlebot or ...


4

It turns out the UserAgent is attached when using the web site monitoring / uptime provider Site Confidence: http://www.siteconfidence.com.au/


4

OMG, are you really keeping your own database up to date? I'm so sorry about you... First tip: If you only need a very simple and minimal solution to detect brand and model, go with WURFL if you want it for free or DeviceAtlas if you can afford it. The second one works probably better (just my personal opinion). Also, take a look at this comparision ...


4

I raised a search enquiry with Yahoo on Monday - the acknowledgement included the following note on the transition to Bing which you might find relevant: Yahoo! Search is excited to reach a new stage in the Yahoo! and Microsoft Search Alliance; the integration of Microsoft algorithmic results is near complete for the U.S. and Canada marketplace. ...


4

Try adding something like the following into your VirtualHost config: SetEnvIf User-agent .*Googlebot.* googlebot CustomLog /var/log/apache2/access_googlebot.log combined env=googlebot CustomLog /var/log/apache2/access.log combined env=!googlebot


4

That's going to block your entire website from being crawled. No There is no such thing as securing your robots.txt. If you don't want to keep visitors out of your directory root you need to prevent that using more secure means. Putting a blank index.html file will easily do the trick. If you're running Apache you can also do it easily using htaccess.


4

Finally realised what the problem was when, after deleting the tag in both files, the error "Invalid command '\xef\xbb\xbf EF BB BF = BOM Moral of the story: Always check your encoding* and hex dump of your files when you get weird issues. *On Notepad++ you can use the Encoding -> Encode in UTF-8 without BOM option to remove it


4

While attempting to block bots can help in free up resources and clean up your logs it’s important to note that robots.txt and even using the meta tag on pages noindex does not actually stop bots visiting your site. They can still crawl your site occasionally to see if the denied from robots has been removed. A lot of bots don’t even use a user agent and ...


3

You failed to share the ip address of the request - so its a bit more difficult to answer the who - but a good idea of why could be a simple scan to check and see if the website is working with all browsers. This is standard for many developers when they make a change - services like www.BrowserShots.org generally show up this way.


3

Completely agree that using a SQL query is not the most effective approach to this. However, if the logs you happen to need are in SQL already, and it's a one off or data sizes a not large enough to warrant improving the approach, here's a handy expression that can be used as a starting point... Based on Sarath's post here: ...


3

Browser manufacturers like to cram unecessary information into their user-agent strings for a variety of reasons. Originally it was to fool browser sniffers that were excluding IE from their website (that's why you see Mozilla/4.0 (compatible; in IE's user-agent). It looks like Microsoft is cramming everything IE8 can be used with into its user-agent which ...


3

Those are from the Google+ app running on the iPad and iPhone. It's running a modified Chrome browser. The mention of Mozilla/5.0 has to do with the history of User Agent Strings, and nothing to do with who built the software. See more here: http://webaim.org/blog/user-agent-string-history/


3

You are already telling them because you have one site using a single set of URLs. They visit twice because each bot evaluates differently. In the end it goes in the same index but the metadata is different which lets them serve different results to standard and mobile queries.


3

Does the position of things in the file matter? No. The groups can be in any order. I'm now a little confused after reading that the addition of User-agent: Googlebot may override certain other values. Any one crawler will only look at one group in the robots.txt file. The crawler will only look at the most specific group, determined by the length ...


3

The user agent belongs to Apache HTTPComponents, which is a Java library that handles HTTP request. For example: It could be an Android app that is using the library to send POST requests to your login script. The UNAVAILABLE part is typically where the version number is located. As far as I know, this user agent is used as the default user agent for ...


2

A better solution IMO would be to detect whether the user is logged in. If they are not, show the standard page (this could be cached). Any web spider will never be logged in but if you are optimizing for them, why not for new users to your site?



Only top voted, non community-wiki answers of a minimum length are eligible