12
votes
Accepted
Spike in Traffic from "Facebot Twitterbot" user agent
This is the iMessages app's crawler (from the users phone itself, not a Apple server).
Apple has chosen to use this useragent in their iMessages app to ensure the unfurling of the URL and the rich ...
- 236
11
votes
Accepted
Was user-agent identification used for some scripting attack techique?
This is a Joomla 0 Day Attack. Information found here: https://blog.sucuri.net/2015/12/remote-command-execution-vulnerability-in-joomla.html
This is not a vulnerability test despite the __test. It is ...
- 32.9k
6
votes
Accepted
What browser is Mozilla/5.0 (Linux; Android 10; SM-A202F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.120 Mobile Safari/537.36?
Below is the syntax for the User Agent header:
User-Agent: Mozilla/5.0 (<system-information>) <platform> (<platform-details>) <extensions>
In your case we have.
Mozilla/5.0 (...
- 6,549
5
votes
Accepted
How did Google, Comscore and Grapeshot learn of a secret page
"It feels like I have spyware on my computer."
Oh, it's much worse than that. There is spyware on everything but your computer, probably.
Grapeshot is an Oracle service. Here are a few ...
- 616
4
votes
Was user-agent identification used for some scripting attack techique?
The IP address that you linked does not resolve to a Google hostname therefore it is not Google. The person or bot is scanning your site for vulnerabilities. The first one is attempting to find a ...
- 33.1k
3
votes
Why do many websites block requests from common HTTP libraries by user-agent?
This is due to the number of people who embed these HTTP libraries in their own software for the purpose of scraping content from other sites, which is often done for the purposes of copyright ...
- 10.2k
2
votes
Accepted
Are user agent names case-sensitive in robots.txt?
Bot names are not case sensitive. However, the filepaths are case sensitive.
Please refer to Google’s Robots.txt Specifications document.
- 56
2
votes
What is the "google-xrawler" user agent used for?
Google requests product feed using the user agent "google-xrawler"
2
votes
Would modifying my website when used by our app with a custom user agent affect SEO?
We did something similar for a couple of mobile apps we built without any obvious detrimental effects to the site's ranking.
We had native apps (Android and iOS), but wanted to show terms and ...
- 2,933
2
votes
Accepted
Why would a user agent string contain both "Samsung" and "AppleWebKit"?
Perhaps they are running Safari on Android https://safariforwindowspc.com/safari-web-browser-android-download/
- 8,556
2
votes
Accepted
HTTP "Vary: User-Agent" header for a Single Page React App that does SSR for robots?
As far as I know Vary header is used for displaying website content via CSS or HTML appropriately for the User Agent who is accessing the site, right? So if your SPA also have different serve between ...
- 1,136
2
votes
Why would a human read robots.txt
Malicious users might have a look at it to determine paths you wanted to hide.
For example
User-Agent: *
Disallow: /admin/
- 121
2
votes
Why would a human read robots.txt
The site owner / marketing people / SEO people may read it to determine if there is a rule blocking access to certain content.
- 121
2
votes
Accepted
For cloaking, what is the difference between the two examples Google gives: changes based on user agent, and text instead of flash to bots?
I don't see the point in the question since cloaking is rather simple, what your users see, is what search engines should see. Analysing what one Google employee has written changes nothing and ...
- 33.1k
2
votes
Was user-agent identification used for some scripting attack techique?
Additionally to other answers, note that the fact that this attack apparently worked suggests you are running an old, insecure version of PHP. A fix for the bug that this attack exploits was released ...
- 131
1
vote
Earliest / easiest to implement way to detect browser, version, and screen size?
First, I think what you should be looking at here is users, sessions, and engaged sessions x traffic from paid channels. You can drill that down by source to understand where exactly they're coming ...
- 6,549
1
vote
Accepted
Would modifying my website when used by our app with a custom user agent affect SEO?
Your app has not only the issue with a quasi-cloaking (displaying slightly different kind of view, as the website does), but with duplicated content too (taking the content from the website). ...
- 10.2k
1
vote
Accepted
SEO Impact of Vary: User-Agent header
After posting the question a day back, I was researching Google official docs in parallel. Fortunately, I found the below phrases in https://developers.google.com/search/mobile-sites/mobile-seo/...
- 2,343
1
vote
Accepted
Requests from a specific older version of Firefox distributed across many Google and Cloudflare IP addresses
I've come to find that this is Discord doing this, as it uses this user agent for fetching images for embeds (maybe the default electron user agent or something similar?), and I can tell as I started ...
- 131
1
vote
What are Windows NT 10.3 and 10.5?
As per List of Microsoft Windows versions wiki there are/were some Windows releases marked as NT 10.0, including Windows 10, Windows Server version 2004, Windows Server version 1909, Windows Server ...
- 123
1
vote
What is the UA "Mozilla/5.0 (compatible; adscanner/)/1.0 (http://seocompany.store; [email protected])" doing on my web server?
That bot developer has not made any information available online publicly (as far as I can tell by searching Google), and the pattern of access you described does not show any specific intent as far ...
- 16.4k
1
vote
How to block user agent using htacces?
If you want to match specific user-agent strings then you don't need to use a regex (in fact, I would recommend not using a regex if you want an exact match). With mod_rewrite, the = CondPattern ...
- 43.1k
1
vote
Accepted
Will human visitors visit through Amazon AWS?
Tough to say without more information, but possibilities you mentioned are likely. They could also be using a VPN and have a DynDNS setup using AWS maybe?
If you check the user behavior and see if it ...
- 915
1
vote
Is "Mozilla/4.0 (compatible;)" a valid user-agent?
It can be a valid user-agent. runlevelsix says this on ServerFault:
According to several resources (such as this entry), these are requests used by proxy tools that employ pre-fetching strategies (...
- 99.4k
1
vote
Could changing the download link and screenshots based on user agent be considered cloaking?
For usability, you should allow users to download the software for a different OS. For example, because the user has multiple OS and wants the software for a different one, or because the one buying ...
- 21.9k
1
vote
What percent of all active IP + User Agent combinations are not unique?
Actually, it has been shown that UserAgent has enough properties that can differ from computer to computer to cause the composite UserAgent string to be unique to few or a single computer/device. To ...
- 11
1
vote
How do you direct iPhone/Android browser to m.example.com?
If you're coding with PHP, you could use $_SERVER['HTTP_USER_AGENT'] for checking the UserAgent header, which is sent almost by any major mobile device to PHP script. Then just compare the value of ...
- 11
Only top scored, non community-wiki answers of a minimum length are eligible