16

According to the list at http://www.useragentstring.com/pages/useragentstring.php?typ=Browser with over 9000 user agent strings from various browsers: 0 user agent strings of browsers contains the word "bot" 2 user agent strings of browsers contains the word "crawl" 0 user agent strings of browsers contains the word "spider" (The 2 which contains "crawl" ...


11

This is a Joomla 0 Day Attack. Information found here: https://blog.sucuri.net/2015/12/remote-command-execution-vulnerability-in-joomla.html This is not a vulnerability test despite the __test. It is an attack. Make sure that any Joomla install is as up to date as possible. Another option is to simply use .htaccess to intercept this exploit by looking for ...


10

Yes. Assuming that your agent names are specified correctly, it looks like this should work. Here is a resource if you want to read more. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt


10

Blocking No User Agent Blocking based on no user agent is a silly idea... a lot of users who like to remain anonymous through VPNS will often disable user agent and anything else that can be used to harvest data... And anonymity is growing. Also if the idea behind this is to save on resources it should be noted that most bots that are not legit search ...


6

Here is a robots.txt file that will allow Google, Bing, and Yahoo to crawl the site while disallowing all other crawling: User-Agent: * Disallow: / User-Agent: googlebot Disallow: User-Agent: bingbot Disallow: User-agent: slurp Disallow: Some crawlers ignore robots.txt entirely and crawl whatever they feel like. Some crawlers impersonate Googlebot or ...


6

Whether or not crawlers honor your robots.txt is entirely an on-your-honor based system. Nothing you put in that file is going to prevent a "fake" crawler from doing anything. With regards to User-agent:, that value is completely voluntary as well. You can instruct your browser, or any other HTTP client to send whatever value you want for that header.


5

Had similar issues with this in the past. From what I've read, there is a big possibility that it is actually IE 11 (really, it comes down to how Webtrends interprets it). This is because IE recently made changes to its User Agent. See here.


5

"It feels like I have spyware on my computer." Oh, it's much worse than that. There is spyware on everything but your computer, probably. Grapeshot is an Oracle service. Here are a few quotes from the crawler explanation page: Oracle Data Cloud assists advertisers placing contextual advertising on web pages. Pages are only visited on demand. If ...


4

There are no rules. A user agent can be anything. There's no reasonable way to whitelist user agents as there are a lot of legitimate ones and you do not want to accidentally block a legitimate user. There's also no way to block bad user agents because, once again, there is no standard way to determine if a user agent represents a bad user. If you want to ...


4

The IP address that you linked does not resolve to a Google hostname therefore it is not Google. The person or bot is scanning your site for vulnerabilities. The first one is attempting to find a Joomla vulnerability. These events are a regular occurrence on most websites, You should ensure that you are following best practices and harden your website, the ...


4

Below is the syntax for the User Agent header: User-Agent: Mozilla/5.0 (<system-information>) <platform> (<platform-details>) <extensions> In your case we have. Mozilla/5.0 (Linux; Android 10; SM-A202F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.101 Mobile Safari/537.36 Still doesn't tell the full picture though, we need ...


3

The space is a delimiter (ie. a special character) in .htaccess so must be backslash escaped if you want to match a literal space in the regex. Eg. DV\ CRAWLER. (Otherwise you are likely to get a less than helpful 500 Internal Server error.) Or, you can use the shorthand character class \s which matches any white space character (space, tab or new line / ...


3

Yes you can. In fact, I recommend it and others too. I would not work so hard on my regular expression to include version numbers- just ^PHP.*$ You will find that some agent names are consistently up to no good even though some will argue that they can be used for good and should not be blocked. That is a negative argument. I study these things and I ...


3

The user agent belongs to Apache HTTPComponents, which is a Java library that handles HTTP request. For example: It could be an Android app that is using the library to send POST requests to your login script. The UNAVAILABLE part is typically where the version number is located. As far as I know, this user agent is used as the default user agent for ...


3

I realize that this is an old post but after researching this I came upon an article explaining the purpose of this user agent. In this article they explain that this is a user agent string used to fetch images in real time when being accessed through the Bing app. Meaning as you scroll through pictures within this app you may end up clicking on one to ...


3

As the user agent is completely client controlled, it is a good thing to pay attention to it, as it can be used in various attacks. Allowed Characters in User Agent what characters are/are not allowed in a proper user-agent string and do some characters need to be in a certain order? @Stephen Ostermiller already linked to RFC2616. It was updated in RFC7231,...


3

If all other traffic from this IP address is legitimate, then I wouldn't worry about the WAF rule being triggered. It doesn't decode into a human readable string. So it was likely inserted by a proxy device for tracking purposes. Regarding your concern about the RFC, they're written as a recommendation for how the field should be used though there is little ...


3

This is due to the number of people who embed these HTTP libraries in their own software for the purpose of scraping content from other sites, which is often done for the purposes of copyright infringement. Well-made crawlers which are legitimate and designed for a specific purpose (like archiver bots and search bots) have their own custom user agent strings ...


2

If a user is using a proxy, there may be headers in the request that you could examine or log to let you know what the original IP address of the user is. See: X-Originating-IP X-Forwarded-For If you are using Apache server, headers such as this can be logged using %{header}i, in the log format configuration directive where "header" can be any HTTP request ...


2

This MSDN article documents the nomenclature used for user agents. Appears the feature token can be used in conjunction with the platform token to make some determinations. Feature tokens Certain optional components can also modify the user-agent string; this next table shows common ones. Be aware that many other applications also modify the user-agent ...


2

Your regular expression specifies that the string must not have anything before or after the name of the crawler. That is what the ^ and $ do. I'm also not sure why you would have a ? in there, which makes the "r" at the end optional. Try this instead: SetEnvIfNoCase User-Agent "siteexplorer" bad_user Deny from env=bad_user


2

It is not clear what your goal is. You do not tell us. If you are trying to identify unique users, you are using two of the most unreliable data elements possible. Yes IP addresses are often used, but in conjunction with other data elements. Here is what you are missing. IP addresses are often dynamic and can change with every session. As well IP addresses ...


2

PHP user agents are known as libraries. It should be no problem if you block them since legitimate crawlers use other string types, however, can't you check and block only the IP's? In this site, for example, you have ways to block only Chinese traffic.


2

Here you get ALL user agents of Yandex bot (this is the Yandex own page, so you shouldn't worry about whether this information's true). Specially about your question's subject: yes, this is a mobile bot. Beside of it, there is another one mobile bot by Yandex. Its user agent is: Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (...


2

Is having a base64-encoded string inside a User-Agent normal or unusual? Digging though the list of User agents at WhichBrowser. It is reasonable to conclude that this is rare, but possibly the result of a malware infection. However I also abused this behaviour to add another security layer to my own site in the past, where only a few clients with a ...


2

Robots.txt Google on occasions has been known to ignore robots.txt, it should never be treated as a guarantee. Sensitive pages should always some of form of authentication, or you could opt to block those user agents by returning a 403 forbidden status. However, this is not the issue. Google Probing Googlebot is known to probe sites with different user ...


2

Additionally to other answers, note that the fact that this attack apparently worked suggests you are running an old, insecure version of PHP. A fix for the bug that this attack exploits was released in september 2015. Run your update process and make sure it pulls in the most recent version of PHP. And check for other outdated programs that are Internet-...


2

Bot names are not case sensitive. However, the filepaths are case sensitive. Please refer to Google’s Robots.txt Specifications document.


2

Perhaps they are running Safari on Android https://safariforwindowspc.com/safari-web-browser-android-download/


2

We did something similar for a couple of mobile apps we built without any obvious detrimental effects to the site's ranking. We had native apps (Android and iOS), but wanted to show terms and conditions, help and privacy pages in-app as well as on the marketing websites for the apps. Rather than duplicating the content across all three we just used the web ...


Only top voted, non community-wiki answers of a minimum length are eligible