11

My website is counting visitor views on certain pages. I noticed that Google and other Bots are "clicking" to my site like crazy and some of the pages get unrealistic view counts (compared to those produced by humans).

I am asking for best practice to exclude those bots from my view counts. Obvious a simple "user agent" contains "bot" won't do it.

I do not think there is a bullet prove solution nor I need one.

Note: I am using PHP + MySQL.

|improve this question
  • The correct answer to your question depends on the kind of software you use for tracking your stats. – Goboozo Apr 5 '11 at 13:06
  • @Goboozo I am using PHP + MySQL. This is not for tracking my stats. This is about queries for "related", "hot" and "interesting" entries. – eisberg Apr 5 '11 at 13:19
18
+100

I'm glad you know there isn't going to be a bulletproof way to accomplish this. That means your outlook is at least realistic.

Since JavaScript is not an option I would say you're left with:

  • Check the user-agent for the word "bot" in it. That will catch most of them.

  • Compile a list of known bots and filter them based on some kind of unique identifier, probably their user-agent.

  • Put a hidden link in the footer of your website that links to a page that collects user-agents and/or IP addresses. Users won't see this but bots will. So anyone who visits that page will be a bot. Record them and then block them from your stats.

|improve this answer
  • This looks promising. I like the idea of a trap. I do something alike for the user comments on my site with an invisible input field (instead of an ugly Captcha). :-) – eisberg Apr 7 '11 at 14:22
  • 2
    @eisberg, It's called a honeypot. I'm a big fan of them myself. – John Conde♦ Apr 11 '11 at 16:09
  • probably best solution... – vkGunasekaran Mar 17 '14 at 7:17
7

There are three fairly simple ways:

  1. Use Google Analytics, which will process and handle all the data for you, and present you with detailed statistics for visitors and how they got to your site. This is by far the easiest solution.
  2. Use Javascript to do the counting. When the page has loaded, generate an AJAX request to your counting script. Robots and spiders don't run Javascript.
  3. Detecting "bot" in the user agent string is actually fairly reliable. Alternatively, you could stick to known bots only such as Googlebot, Yahoo, MSNbot etc. Checking those three should cover 99% of your bot traffic. This page has some others but it looks quite out of date.

UPDATE: Googlebot and some major bots do run JavaScript these days. So using option #2 alone is no longer viable. However, this does mean using it in conjunction with #3 should be quite reliable, as you can easily exclude most bots by using JS, then on the server-side exclude major bots like Googlebot that do run JS.

Also as mentioned in the comments you could try using the Google Analytics API to display views for each page.

|improve this answer
  • 1 and 2 are unrelated to my question. Maybe I should have asked "How do I get a view counter like stackechange which ignores bots?" so my false :-) Nonetheless very nice list in 3. Thank you I will check this. – eisberg Apr 5 '11 at 17:48
  • Sorry, didn't realise you were displaying the view count on the page. In that case 2 and 3 are both viable. – DisgruntledGoat Apr 5 '11 at 23:50
  • Thank you for the effort. At least for me javascript is never the answer so I am stuck with 3 but there are so many indicators :-( – eisberg Apr 6 '11 at 7:06
  • Actually, some bots do run some JavaScript. I set up an Ajax request to track viewport size in relation to screen size. Googlebot is reporting two different screen sizes. – toxalot Nov 15 '13 at 23:53
  • I know the question is old and has already a solution but why not use Google Analitycs API if it's only to show a human views counter? stackoverflow.com/questions/19484009/… – KeizerBridge Apr 9 '16 at 12:26
3

If you use Javascript to count views then most bots won't run it and so won't be included in your view counts. This answer may be close to what you want https://stackoverflow.com/questions/1973448/how-can-i-count-a-page-views

|improve this answer
  • No I am not using Javascript. It is a normal PHP + MySQL website. But thanks for the reply anyway :-) – eisberg Apr 5 '11 at 13:40
  • Don't discount this answer so fast. Even with a site using PHP+MySQL, nothing stops you from emitting a bit of JavaScript to generate your view counter. Instead of outputting the counter directly, output something like '<script>document.write(<counter-code>)</script>' where <counter-code> is whatever you need to create your counter. – Itai Apr 5 '11 at 19:37
  • @Itai Sorry, but I won't use javascript to count my visitors. But thank you anyway. – eisberg Apr 6 '11 at 7:07
  • If you don't want to use JavaScript, just put it in your question clearly. As Itai said, nothing prevent us from thinking about JavaScript solution for a PHP site. – Tien Do Feb 7 '12 at 2:31
  • Actually, some bots do run some JavaScript. I set up an Ajax request to track viewport size in relation to screen size. Googlebot is reporting two different screen sizes. – toxalot Nov 15 '13 at 23:56
3

My approach involves two passes:

  1. Filter only web browsers and consoles by matching the start of the user agent string with Mozilla|Opera|PSP|Bunjalloo|wii. Thanks to the user agent spoofing this check will detect almost all browsers
  2. Exclude bots by common stop strings bot|crawl|slurp|spider

So if the first step is passed we assume that it is a browser and there is a real visitor behind it. As I found out though some bots pretend to be Mozilla compatible and start their user agent string with it. That's why the second pass might come in handy and eliminate them.

function isBrowser () {

    return preg_match( '/^(Mozilla|Opera|PSP|Bunjalloo|wii)/i', $_SERVER['HTTP_USER_AGENT'] ) && !preg_match( '/bot|crawl|slurp|spider/i', $_SERVER['HTTP_USER_AGENT'] );
}
|improve this answer
2

I use just simple user agent parsing exclusion. It gets rid of 99% of bots going into my pages.

SELECT * FROM `live_visitors` where (
lower(agent) != '%bot%' and
lower(agent) != '%slurp%' and
lower(agent) != '%spider%' and
lower(agent) != '%crawl%' and
lower(agent) != '%archiver%' and
lower(agent) != '%facebook%')
|improve this answer
  • 1
    This would not help (at least in my case) because almost every bot I can identify by its behavior (too fast browsing, chronological browsing of all links, ...) uses a valid User-Agent. But good idea for other projects. – eisberg Jun 7 '11 at 9:21
2

You can use an image as a counter, in this case it won't count bots and page name is passed as a query with image name

I'm using this at img.php which update page view in database:

<?php
$xnt = $_GET["ID"]; if (isset($xnt) && is_numeric($xnt)) {
$DBServer = "localhost"; $DBUser = "xxx"; $DBPass = "xxx"; $DBaze = "xxx";
$conn = mysqli_connect($DBServer, $DBUser, $DBPass, $DBaze); 
mysqli_query($conn, "UPDATE stats SET stats_vz=stats_vz+1 WHERE stats_id=".$xnt);
mysqli_close($conn);}
$im = @imagecreatetruecolor(1, 1); imagesavealpha($im, true); imagealphablending($im, false); $white = imagecolorallocatealpha($im, 255, 255, 255, 127); imagefill($im, 0, 0, $white);
header("Content-type: image/png"); imagepng($im); imagedestroy($im);
|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.