Hot answers tagged

58

GoAccess is a free text/curses based log analyzer similar to "top".


12

This is Vincent from Bing Webmaster Tools. This was a Bingbot checking for an XML sitemap generated by the Bing Sitemap Plugin (Beta) for IIS and Apache - see the following for more details: Bing Sitemap Plugin (Beta) - Bing Webmaster Tools Help & How-To Center Building Sitemaps Manually?… Stop Until You Read This! - Bing Webmaster Blog You shouldn't ...


10

Matomo (former Piwik) is a good choice (better than awstats, in my opinion)


5

If you want to analyze only "human traffic" I would not count the ones with empty or missing user agent string. In my experience almost any browser will always send one. Even most privacy plugins or extensions rather fake (include other OS or Client name) or "normalize" (e.g. no release numbers) or randomize (e.g. sometimes FF, sometimes IE strings) the UA ...


5

This is just referer spam, from Wikipedia: Referrer spam (also known as log spam or referrer bombing) is a kind of spamdexing (spamming aimed at search engines). The technique involves making repeated web site requests using a fake referer URL to the site the spammer wishes to advertise. Sites that publish their access logs, including referer statistics, ...


4

Completely agree that using a SQL query is not the most effective approach to this. However, if the logs you happen to need are in SQL already, and it's a one off or data sizes a not large enough to warrant improving the approach, here's a handy expression that can be used as a starting point... Based on Sarath's post here: http://www.consultsarath.com/...


4

Your first log snippet suggests that the file may actually exist, since nginx response is 200, with 731 bytes of payload data, so, don't be too fast to have it blocked. However, if you want to have such requests omitted from logs, you can use the following in all servers affected: location = /wpad.dat { access_log off; log_not_found off; } Or, ...


3

MixPanel is reasonably priced, has an excellent feature set and has emailed reports.


3

I would be suspicious: 208.115.113.90 - - [09/Aug/2012:00:58:17 -0400] "GET /?p=unauthmanagement HTTP/1.0" 200 8998 "-" "Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)" ‹ùU#Pmoo.tigertutorialscom First: GET /?p=unauthmanagement What is this? Does this page exist? "unauthorized management" could be a script that someone uploaded to do malicious ...


3

Any type of logging software such as Awstats and Webalizer do not contain the information you are after, you should take a look at various Analytics platforms and see what one is best suited for your business. I personally recommend Google Analytics but there are many more out their to test. Please note that all links listed below and are in no particular ...


3

That kind of information is collected for website usability testing. There is a strong correlation between where the mouse moves and where users read (many users move the mouse near what they are reading). Knowing where the mouse goes and what users click on gives webmasters a good sense of what users find important and what most draws their attention on ...


2

I'm going to add another answer here too. Based on the Google Analytics documentation for their iOS and Android SDN, it's possible to send traffic events from non-webpages. Some people have figured out how to use this API to send traffic directly from your server-side server-side application code, so now you can use the Google Analytics API to send traffic ...


2

Another one that I've noticed is browserconfig.xml which is some bastardisation requested by IE. Bloody rubbish filling up the logs.


2

If you're looking for an offline analyzer, I quite like Sawmill: https://www.sawmill.co.uk It analyses all sorts of logs, including apache. Has a free trial period.


2

I work for a security company and among other things we monitor Bad Bot traffic. Based on my experience, humans visits with blank user-agent data indicate scraping/spamming attempts (usually scraping) made by "headless browser" bots. These visitors can sometimes execute JS, and so they will appear in GA - still, this dose not make them human :) Apologize ...


2

Make sure that you don't have log specified - from the documentation on ModSecurity: Description: Indicates that a successful match of the rule needs to be logged. Example: SecAction phase:1,id:117,pass,initcol:ip=%{REMOTE_ADDR},log This action will log matches to the Apache error log file and the ModSecurity audit log. Instead specify ...


2

It is common practice to define a seperate logfile for each vhost. Without knowing your config, I assume, that the main nginx.conf has defined a default log file. It will log any request that is not matching any of the vhosts that might be configured. I assume that sites-enabled/mysite is a vhost that is using name-based virtual host (in Apache speech this ...


2

Apache supports rotating logfiles. Check the manual here, specifically the following example: CustomLog "|bin/rotatelogs -l /var/logs/logfile.%Y.%m.%d 86400" common


2

Yes, this bot is trying to exploit JCE. It’s probably using a very powerful publicly available exploit (+ mirror) for JCE < 2.0.11. Although this is a really outdated version, many instances are still in the wild. JCE 2.0.11 was released on 2011-08-29, the exploit has been available after the release (if its author kept his word). If JCE is not installed ...


2

If you wish to disable the access logging, you will need access to httpd.conf file. This is possible only if you have root privileges. I won't be possible to edit these values with cPanel access.


2

Unfortunately it is not possible to disable or change the access logging configuration from within cPanel, or using .htaccess files or any other tricks - this can only be done with server administration privileges, not a single cPanel user privileges. Greater control over the logging would be a great feature to add into cPanel, though I can't personally ...


2

I have hundreds of sub-domains of opera-mini.net. Too many to count! I did a quick review of about 20 of them and I see two things: 1] Rogue Spider: A spider or bot that disobeys or ignores the robots.txt file. 2] Stealth Spider: A spider or bot that tries to fly under the site owners radar when analyzing their log file. It is likely that this is a ...


2

Are you displaying ads on your site from Amobee (formerly known as Kontera)? The PTR record for this IP is nat.aws.kontera.com, suggesting that it is a crawler looking for your page's content in order to determine what ads may be relevant to that URL. If you've loaded the Kontera JavaScript on login-protected pages, then you will find that they are crawled ...


2

You can use something like this 1) Set up onBeforeAction hooks to login the user automatically (which asks for credentials if user is not logged in to external service) var loginWithGoogle = function() { if (bla.isClient) { Session.set('loginError', undefined); bla.loginWithGoogle({ loginStyle : "redirect", requestPermissions :...


2

As @dan hinted, your problem is probably not related to logging, as Apache would cheerfully also log IPv6 addresses... if someone connects to it over IPv6. So you will first need to double check that your server is indeed listening on an IPv6 and is reachable from outside. For that, you may need the Listen directive. Try locally by going to http://[::1]/, ...


2

You could certainly use something like ab or siege -- but it seems like a lot of work for little benefit, at least given what you're trying to accomplish. If I were you I'd just create some random/empty files. Give them the right name and put them in the right place and logrotate should do its job normally -- it doesn't care at all about what the contents of ...


1

Unless your hosts built in web analytics software will break down page views for you, there won't be a great deal you can do if I follow correctly (and this will vary depending on host).. Would it not be possible to import the raw access logs into MS Excel or similar and then use a program such as Xenu to crawl your site. Export the list of URL's from your ...


1

A good method is what Stephen posted but in additional to that I believe Google Analytics can be setup to pretty much any 'Event' using their API. I'm sure this is something you can get a web developer to code but then you would need to host the files on your own hosting such as a CDN since on a normal server it'll drain your bandwidth with large mp3's. It ...


1

Have you looking into SoundCloud? I noticed that Stack Exchange uses them to host their podcast files. They have an article about how to use the statistics they collect.


Only top voted, non community-wiki answers of a minimum length are eligible