Why does StatCounter show less unique visitors than my Apache access log?

Question

StatCounter shows yesterday as seeing 10,611 unique visitors to my site.

I wrote a script that counted the number of unique IP addresses in yesterday's Apache access log. It counted 14,190 unique IP addresses.

Why is there the discrepancy? I would understand a small number of IP addresses would maybe be for crawlers and bots, but not 1/3 of the traffic.

// script to count unique IPs

<?php
$ips = array();

$file = fopen("access_log","r");
while ( ($line = fgets($file)) != false )
{
    $num = strpos($line, " ");
    $ip =  substr($line, 0, $num)."<br>";

    if ( !in_array($ip, $ips) )
        array_push($ips, $ip);
}
fclose($file);

echo count($ips);

The StatCounter code is the standard code that StatCounter gives you to put in your HTML/JS, which includes your account number, etc...

Answer 1

Providing initial explanation re difference between server-side stats and online stats for the benefit of other users:

Apache access log analyses all the traffic to your server.

StatCounter tracks all the visits to your site.

Your server gets a lot more traffic than your site, because server traffic includes all the visits from robots, spambots, crawlers etc.

When a robot visits your site, this is recorded in your server logs. It's not a 'true' visit as the page is not fully loaded, so it doesn't show up on StatCounter.

So server-side stats (e.g. Apache log files) will always appear inflated compared to online stats (like StatCounter). Generally it is the online stats, the 'true' visits that people are interested in - it's these visits that might mean more sales, more sign ups etc. However, the Apache log files are obviously very useful from a more technical standpoint and for server administration. If you exclude the bot and crawler hits from your Apache log - then what you're left with should approximately mirror the StatCounter numbers.

---

Now - in relation to you specific case, in our experience, 30% more traffic from crawlers and bots is not unusual.

I should also add that Unique Visitors in the StatCounter summary stats are based on cookies (rather than IP addresses) which will also explain some of the difference.

AOL visitors can have a different IP with every pageload - so that will drive IP based uniques higher with every pageload. OR users who disallow cookies will drive the cookie based uniques higher with every pageload.

In StatCounter, visitor paths are collated using IP addresses so that may be a more accurate comparison for you. Drop us a line if we can help further.

Answer 2

many reason can cause this mismatch, for example:

1. you use server-side counter which counts each access immediately. statcounter use javascript which count access after page load. so some access can be counted by your script but not with statcounter
2. what is the type of your website? I had this experience in forums and blogs which made by popular applications like phpBB and Drupal
3. where you put your script and where statcounter's? are you sure both are counting same pages?
4. ...

Answer 3

Counting number of unique IP address is probably not the best approach, as it will include robots and they hit your server from different IP's number of times during the day.

1. I use awstats on my server as it clearly differentiates between robots and actual users, so gives you a fair idea about traffic.
2. In fact using awstats alone might not give clear picture, it's always better to use google analytics or statcounter which tracks users using javascript. awstats rely only upon unique IPaddress to identify unique users BUT most of the offices where there are hundreds of people working has only one outgoing IPaddress,so javascript based trackers like statcounter or google analytics are more reliable here as they identify unique user by using a cookie on the user browser.
3. I actually use all three awstats , statcounter and google analytics as each one of them provides interesting aspects about different things. Page loading isn't a issue for me, i have anyhow included javascript code for tacking as a seperate JS file and not embedded on html which ensures it is cached on client side.