15

According to the list at http://www.useragentstring.com/pages/useragentstring.php?typ=Browser with over 9000 user agent strings from various browsers: 0 user agent strings of browsers contains the word "bot" 2 user agent strings of browsers contains the word "crawl" 0 user agent strings of browsers contains the word "spider" (The 2 which contains "crawl" ...


11

This is a Joomla 0 Day Attack. Information found here: https://blog.sucuri.net/2015/12/remote-command-execution-vulnerability-in-joomla.html This is not a vulnerability test despite the __test. It is an attack. Make sure that any Joomla install is as up to date as possible. Another option is to simply use .htaccess to intercept this exploit by looking for ...


10

Yes. Assuming that your agent names are specified correctly, it looks like this should work. Here is a resource if you want to read more. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt


10

Blocking No User Agent Blocking based on no user agent is a silly idea... a lot of users who like to remain anonymous through VPNS will often disable user agent and anything else that can be used to harvest data... And anonymity is growing. Also if the idea behind this is to save on resources it should be noted that most bots that are not legit search ...


7

The first usage of this I could find was with the Heritrix crawler. In this manual document, I found the following: 6.3.1.3.2. user-agent The initial user-agent template you see when you first start heritrix will look something like the following: Mozilla/5.0 (compatible; heritrix/0.11.0 +PROJECT_URL_HERE You must change at least the ...


6

Here is a robots.txt file that will allow Google, Bing, and Yahoo to crawl the site while disallowing all other crawling: User-Agent: * Disallow: / User-Agent: googlebot Disallow: User-Agent: bingbot Disallow: User-agent: slurp Disallow: Some crawlers ignore robots.txt entirely and crawl whatever they feel like. Some crawlers impersonate Googlebot or ...


6

Whether or not crawlers honor your robots.txt is entirely an on-your-honor based system. Nothing you put in that file is going to prevent a "fake" crawler from doing anything. With regards to User-agent:, that value is completely voluntary as well. You can instruct your browser, or any other HTTP client to send whatever value you want for that header.


5

I downloaded all the user agents from http://www.user-agents.org/ and ran a script to count the number of them that used the + style links vs plain links. I excluded the "non-standard" user agent strings that don't match RFC 2616. Here are the results: Total: 2471 Standard: 2064 Non-standard: 407 No link: 1391 With link: 673 Plus link: 145 Plain link: ...


5

Had similar issues with this in the past. From what I've read, there is a big possibility that it is actually IE 11 (really, it comes down to how Webtrends interprets it). This is because IE recently made changes to its User Agent. See here.


5

"It feels like I have spyware on my computer." Oh, it's much worse than that. There is spyware on everything but your computer, probably. Grapeshot is an Oracle service. Here are a few quotes from the crawler explanation page: Oracle Data Cloud assists advertisers placing contextual advertising on web pages. Pages are only visited on demand. If ...


4

Does the position of things in the file matter? No. The groups can be in any order. I'm now a little confused after reading that the addition of User-agent: Googlebot may override certain other values. Any one crawler will only look at one group in the robots.txt file. The crawler will only look at the most specific group, determined by the length of ...


4

There are no rules. A user agent can be anything. There's no reasonable way to whitelist user agents as there are a lot of legitimate ones and you do not want to accidentally block a legitimate user. There's also no way to block bad user agents because, once again, there is no standard way to determine if a user agent represents a bad user. If you want to ...


4

The IP address that you linked does not resolve to a Google hostname therefore it is not Google. The person or bot is scanning your site for vulnerabilities. The first one is attempting to find a Joomla vulnerability. These events are a regular occurrence on most websites, You should ensure that you are following best practices and harden your website, the ...


3

The user agent belongs to Apache HTTPComponents, which is a Java library that handles HTTP request. For example: It could be an Android app that is using the library to send POST requests to your login script. The UNAVAILABLE part is typically where the version number is located. As far as I know, this user agent is used as the default user agent for ...


3

You are already telling them because you have one site using a single set of URLs. They visit twice because each bot evaluates differently. In the end it goes in the same index but the metadata is different which lets them serve different results to standard and mobile queries.


3

I realize that this is an old post but after researching this I came upon an article explaining the purpose of this user agent. In this article they explain that this is a user agent string used to fetch images in real time when being accessed through the Bing app. Meaning as you scroll through pictures within this app you may end up clicking on one to ...


3

Yes you can. In fact, I recommend it and others too. I would not work so hard on my regular expression to include version numbers- just ^PHP.*$ You will find that some agent names are consistently up to no good even though some will argue that they can be used for good and should not be blocked. That is a negative argument. I study these things and I ...


3

The space is a delimiter (ie. a special character) in .htaccess so must be backslash escaped if you want to match a literal space in the regex. Eg. DV\ CRAWLER. (Otherwise you are likely to get a less than helpful 500 Internal Server error.) Or, you can use the shorthand character class \s which matches any white space character (space, tab or new line / ...


3

If all other traffic from this IP address is legitimate, then I wouldn't worry about the WAF rule being triggered. It doesn't decode into a human readable string. So it was likely inserted by a proxy device for tracking purposes. Regarding your concern about the RFC, they're written as a recommendation for how the field should be used though there is little ...


3

This is due to the number of people who embed these HTTP libraries in their own software for the purpose of scraping content from other sites, which is often done for the purposes of copyright infringement. Well-made crawlers which are legitimate and designed for a specific purpose (like archiver bots and search bots) have their own custom user agent strings ...


2

As Gisle Hannemyr mentioned in a comment, the best way to do this is to require logins of all users, and do not provide the restricted content to anyone who isn't logged in. If you can't require logins for some reason, there are still a couple of fallbacks you can use (disclaimer: both of them are either partly or completely my fault): The OWASP ...


2

If a user is using a proxy, there may be headers in the request that you could examine or log to let you know what the original IP address of the user is. See: X-Originating-IP X-Forwarded-For If you are using Apache server, headers such as this can be logged using %{header}i, in the log format configuration directive where "header" can be any HTTP request ...


2

This MSDN article documents the nomenclature used for user agents. Appears the feature token can be used in conjunction with the platform token to make some determinations. Feature tokens Certain optional components can also modify the user-agent string; this next table shows common ones. Be aware that many other applications also modify the user-...


2

Your regular expression specifies that the string must not have anything before or after the name of the crawler. That is what the ^ and $ do. I'm also not sure why you would have a ? in there, which makes the "r" at the end optional. Try this instead: SetEnvIfNoCase User-Agent "siteexplorer" bad_user Deny from env=bad_user


2

It is not clear what your goal is. You do not tell us. If you are trying to identify unique users, you are using two of the most unreliable data elements possible. Yes IP addresses are often used, but in conjunction with other data elements. Here is what you are missing. IP addresses are often dynamic and can change with every session. As well IP addresses ...


2

PHP user agents are known as libraries. It should be no problem if you block them since legitimate crawlers use other string types, however, can't you check and block only the IP's? In this site, for example, you have ways to block only Chinese traffic.


2

Here you get ALL user agents of Yandex bot (this is the Yandex own page, so you shouldn't worry about whether this information's true). Specially about your question's subject: yes, this is a mobile bot. Beside of it, there is another one mobile bot by Yandex. Its user agent is: Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (...


2

As the user agent is completely client controlled, it is a good thing to pay attention to it, as it can be used in various attacks. Allowed Characters in User Agent what characters are/are not allowed in a proper user-agent string and do some characters need to be in a certain order? @Stephen Ostermiller already linked to RFC2616. It was updated in ...


2

Is having a base64-encoded string inside a User-Agent normal or unusual? Digging though the list of User agents at WhichBrowser. It is reasonable to conclude that this is rare, but possibly the result of a malware infection. However I also abused this behaviour to add another security layer to my own site in the past, where only a few clients with a ...


2

Robots.txt Google on occasions has been known to ignore robots.txt, it should never be treated as a guarantee. Sensitive pages should always some of form of authentication, or you could opt to block those user agents by returning a 403 forbidden status. However, this is not the issue. Google Probing Googlebot is known to probe sites with different user ...


Only top voted, non community-wiki answers of a minimum length are eligible