2

I used this line to extract unique IPs from my access_log for the last minute:

grep "2017:19:23" /var/log/nginx/access_log | awk '{print $2}' | sort -n | uniq -c | sort -nr | head -20000 | wc -l

Result: 185

I then tried again for 10 seconds:

grep "2017:19:23:0" /var/log/nginx/access_log | awk '{print $2}' | sort -n | uniq -c | sort -nr | head -20000 | wc -l

Result: 74

What does google analytics realtime tell me? 37. I've not seen it go over 50 all day and I'm confident that at time my concurrent users has been more like 200+.

Any idea why this would be?

||improve this question
  • Google counts what it feels are real users and not bots. While a bot can look like a user in the log file, Google tracks many billions of accesses and can better determine non-users better than we can. Cheers!! – closetnoc Aug 7 '17 at 18:32
  • I've solved it. I had cookieDomain set to my-domain-here.com. I've changed it to www.my-domain-here.com now and suddenly my real time users are up to 157. Weird that that "bug" would affect some hits but not others - I'd understand if it had said "0"! – Codemonkey Aug 7 '17 at 18:39
  • GA doesn't count bots that don't use JavaScript. Like 50% of internet traffic is bots. – Stephen Ostermiller♦ Aug 7 '17 at 20:28
  • Thanks @StephenOstermiller, but in this case that wasn't my issue, or not my main issue at least! – Codemonkey Aug 8 '17 at 7:03
1

I've solved it.

I had cookieDomain set to https://www.my-domain-here.com. I've changed it to www.my-domain-here.com now and suddenly my real time users are up to 157.

Weird that that "bug" would affect some hits but not others - I'd understand if it had said "0"!

||improve this answer
  • The distressing thing is that this has been wrong for 10 months, so all of my ga data for that time is rubbish :( I thought it was odd that sales were up while traffic was down! – Codemonkey Aug 7 '17 at 18:45
  • I would use the fact that sales were up while traffic was down to tout your skills as a remarkable SEO and should therefore get a raise! You can claim more relevant traffic. It is all about perspective my Man! (humor... in case no-one figures that out) Cheers!! – closetnoc Aug 7 '17 at 18:50
  • Self-employed sole dev :) – Codemonkey Aug 7 '17 at 19:35
  • I have always been self employed too and my boss has always been a royal pain in my @$$. Want a jerk! I never get time off when I want, I do not get paid enough, and everytime I quit, I end up back the next day! What is up with that?? – closetnoc Aug 7 '17 at 20:29
0

...extract unique IPs from my access_log... What does google analytics realtime tell me? 37. I've not seen it go over 50 all day and I'm confident that at time my concurrent users has been more like 200+.

It's all about fine timing. Let's just assume the data you want to examime and the data google wants to examine is the same:

What you have demonstrated with your code is that the log is generated on the same server that you're querying accurate results from. The time to access a log locally is in the order of microseconds or less. It will even be faster if the log file was stored on a ram drive.

Now what google (or any other remote service) needs to do is connect to the internet in order to access a remote item (such as your log). Factor the time it takes for google to connect and factor your internet packet processing settings on your server (example: tcp/ip settings and packet priority settings and the like) then you can add that time to the time it takes for the local log file to be accessed. A normal amount for this time would be in the order of milliseconds which is 100x slower compared to accessing the file locally.

Now if google decided to download an entire log instead of requesting a specific entry, then the timing will be way off since time has to be spent downloading a bunch of entries only to get one.

||improve this answer
  • Why would google be accessing my log file, that's not how this works, that's not how any of this works! – Codemonkey Aug 8 '17 at 7:07
  • FWIW though I answered my own question, look above. – Codemonkey Aug 8 '17 at 7:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.