3

I wrote my own statistics logging script. It excludes a lot of bots - ie any user-agent that contains any of the following is excluded:

bot
spider
crawl
saashub
owler
researchscan
go-http-client
newsblur
python-requests
alittle client
hypefactors
dalvik
duckduckgo
electronic frontier foundation
lwp::simple
lenns.io
magellan
qwant
yahoo
saashub
cfnetwork
curl
facebook
montastic

I would imagine that there are lots more but not enough to make a massive impact.

Anyway - this script of mine is just a PHP function that writes a line to a logfile. The logfile is then parsed when I want to see the stats.

The thing is - it reports an awful lot more than Google Analytics did - over 2000 for my script vs a few hundred for GA. Is there something I'm missing?

Improve this question

1 Answer 1

Reset to default
5

There could be quite a bit you are missing - but without knowing details of your script we can't say how much.

One of the key things that may largely describe the difference is that GA uses a bit of Javascript to count hits. Bots generally don't run Javascript - and there are a lot of bots not on your list.

Conversely, I don't believe Google accounts for when I visit a web page - because I have adblocking. While I am not your typical user, I am by no means unique. Your script would pick up my hit (correctly), but Google won't.

There may also be discrepancies related to caching, and what you consider a hit.

Your mileage will likely vary a lot - but I did a very quick and dirty list of agents hitting my customer home pages over the last few hours (almost all using Wordpress which will skew things quite a bit but) - and you missed a few very heavily traffic'd bots including "FreshpingBot" at over 10% of hits relative to Mozilla hits, Sogou web pider, Wordpress, Apache-HttpClient, WP Fastest Cache Preload bot, ZoominfoBot, cpp-httpli, mediatoolkitbot, webprosbot,Ruby, EPiServer Link Checker, panscient.com stood out among many others I saw not on your list.

In short, Googles results are likely more accurate then your results if you care about real users, but they likely slightly understate the number hits.

Improve this answer
10
  • Thanks for the answer. Do you happen to know of a comprehensive plain text list of user agents that I could exclude? A few on your list (those with bot or spider in the name) will be caught by my checks (ie partial matches on "bot" or "spider").
    – Richard
    Nov 18 at 22:12
  • 1
    I don't know of a comprehensive list, and I expect that it won't be comprehensive for an extended period. Also what about fake bots masquerading as a common user agent? Not sure if you are aware of a piece of software called "Webalizer". It looks to be doing the kind of thing you are interested in. Problem is that although it was free, ubiuqitous and opensource it looks like it was last updated 9 years ago. My guess is it is not maintained for a number of reasons - but one of which is its a game of whack-a-mole you can't win. (I went looking to it to see if it had a list of bots)
    – davidgo
    Nov 18 at 22:21
  • No idea how old it is, but have you seen user-agents.net/bots ? Also explore.whatismybrowser.com/useragents/explore - Its interesting that explore.whatismybrowser.com/useragents/explore/… shows a large heap of bots which are identified starting with Moizlla/5.0...
    – davidgo
    Nov 18 at 22:27
  • 1
    Thats not stupid. My guess is it would help a lot. I comment you probably only need to pass the referer as the ajax request will still come ftom the browser so should have the IP and user agent. My guess is a post request might be marginally more accurate.
    – davidgo
    Nov 20 at 18:00
  • 1
    Now, after a week(ish), it seems to be going well. The script that parses the logs is surprisingly quick too even when looking at a near months worth of logs. The POST AJAX thing seems to be cutting out a lot of junk - and from what's being logged I could also add "anything that doesn't start with Mozilla/ can be dumped" since all of the valid referers seem to. I do worry about performance though if the site were ever to get busy (for example yesterday was a good day and I only got just over 700 requests logged) but I could optimise the logging script a bit if that ever happens. Cheers.
    – Richard
    Nov 30 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.