Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

My website is counting visitor views on certain pages. I noticed that Google and other Bots are "clicking" to my site like crazy and some of the pages get unrealistic view counts (compared to those produced by humans).

I am asking for best practice to exclude those bots from my view counts. Obvious a simple "user agent" contains "bot" won't do it.

I do not think there is a bullet prove solution nor I need one.

Note: I am using PHP + MySQL.

share|improve this question
    
The correct answer to your question depends on the kind of software you use for tracking your stats. –  Goboozo Apr 5 '11 at 13:06
    
@Goboozo I am using PHP + MySQL. This is not for tracking my stats. This is about queries for "related", "hot" and "interesting" entries. –  eisberg Apr 5 '11 at 13:19

4 Answers 4

up vote 9 down vote accepted
+100

I'm glad you know there isn't going to be a bulletproof way to accomplish this. That means your outlook is at least realistic.

Since JavaScript is not an option I would say you're left with:

  • Check the user-agent for the word "bot" in it. That will catch most of them.

  • Compile a list of known bots and filter them based on some kind of unique identifier, probably their user-agent.

  • Put a hidden link in the footer of your website that links to a page that collects user-agents and/or IP addresses. Users won't see this but bots will. So anyone who visits that page will be a bot. Record them and then block them from your stats.

share|improve this answer
    
This looks promising. I like the idea of a trap. I do something alike for the user comments on my site with an invisible input field (instead of an ugly Captcha). :-) –  eisberg Apr 7 '11 at 14:22
1  
@eisberg, It's called a honeypot. I'm a big fan of them myself. –  John Conde Apr 11 '11 at 16:09
    
probably best solution... –  Sekar Mar 17 at 7:17

I use just simple user agent parsing exclusion. It gets rid of 99% of bots going into my pages.

SELECT * FROM `live_visitors` where (
lower(agent) != '%bot%' and
lower(agent) != '%slurp%' and
lower(agent) != '%spider%' and
lower(agent) != '%crawl%' and
lower(agent) != '%archiver%' and
lower(agent) != '%facebook%')
share|improve this answer
1  
This would not help (at least in my case) because almost every bot I can identify by its behavior (too fast browsing, chronological browsing of all links, ...) uses a valid User-Agent. But good idea for other projects. –  eisberg Jun 7 '11 at 9:21

If you use Javascript to count views then most bots won't run it and so won't be included in your view counts. This answer may be close to what you want http://stackoverflow.com/questions/1973448/how-can-i-count-a-page-views

share|improve this answer
    
No I am not using Javascript. It is a normal PHP + MySQL website. But thanks for the reply anyway :-) –  eisberg Apr 5 '11 at 13:40
    
Don't discount this answer so fast. Even with a site using PHP+MySQL, nothing stops you from emitting a bit of JavaScript to generate your view counter. Instead of outputting the counter directly, output something like '<script>document.write(<counter-code>)</script>' where <counter-code> is whatever you need to create your counter. –  Itai Apr 5 '11 at 19:37
    
@Itai Sorry, but I won't use javascript to count my visitors. But thank you anyway. –  eisberg Apr 6 '11 at 7:07
    
If you don't want to use JavaScript, just put it in your question clearly. As Itai said, nothing prevent us from thinking about JavaScript solution for a PHP site. –  Tien Do Feb 7 '12 at 2:31
    
Actually, some bots do run some JavaScript. I set up an Ajax request to track viewport size in relation to screen size. Googlebot is reporting two different screen sizes. –  toxalot Nov 15 '13 at 23:56

There are three fairly simple ways:

  1. Use Google Analytics, which will process and handle all the data for you, and present you with detailed statistics for visitors and how they got to your site. This is by far the easiest solution.
  2. Use Javascript to do the counting. When the page has loaded, generate an AJAX request to your counting script. Robots and spiders don't run Javascript.
  3. Detecting "bot" in the user agent string is actually fairly reliable. Alternatively, you could stick to known bots only such as Googlebot, Yahoo, MSNbot etc. Checking those three should cover 99% of your bot traffic. This page has some others but it looks quite out of date.
share|improve this answer
    
1 and 2 are unrelated to my question. Maybe I should have asked "How do I get a view counter like stackechange which ignores bots?" so my false :-) Nonetheless very nice list in 3. Thank you I will check this. –  eisberg Apr 5 '11 at 17:48
    
Sorry, didn't realise you were displaying the view count on the page. In that case 2 and 3 are both viable. –  DisgruntledGoat Apr 5 '11 at 23:50
    
Thank you for the effort. At least for me javascript is never the answer so I am stuck with 3 but there are so many indicators :-( –  eisberg Apr 6 '11 at 7:06
    
Actually, some bots do run some JavaScript. I set up an Ajax request to track viewport size in relation to screen size. Googlebot is reporting two different screen sizes. –  toxalot Nov 15 '13 at 23:53

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.