0

I'm recently doing keyword advertising on Bing, which also powers AOL and Yahoo searches, to promote a product. And I'm using a third-party tracker to track the clicks of my advertisements.

I found that the third-party tracker I'm using efficiently isolates all "clicks" (not sure if those are really "clicks") made by various search engine bots/crawlers into a different result group (temporary screenshot). I didn't even heard of some of those bots/crawlers before. That bot-identifying feature is very useful in the analysis of performance of my advertisments.

And I'm planning to create my own tracker so that I don't have to rely on a third-party tracker in the future. Therefore I need to know how to identify all of those search engine bots/crawlers in my to-be-built tracker.

|improve this question
  • 1
    You can use Fail2Ban to block bad bots easily. Otherwise determining a good bot from a bad bot, while easy in that they behave differently, is fairly technical and not advised. I should know. It was my business for a long time. In this answer I outline some of what makes a bad bot detectable. webmasters.stackexchange.com/questions/67221/… I recommend blocking using a well managed product such as Fail2Ban instead of rolling your own. Cheers!! – closetnoc Apr 16 '18 at 6:27
  • @closetnoc Thanks. I just checked Fail2Ban. It is written in Python. Can it be installed on my local server and remote server (Bluehost)? – Ian Y. Apr 16 '18 at 7:30
  • 1
    I can't t tell you if python is installed on your systems. However it likely can be quite easily and without issue. Python has become a fairly standard language these days. Here is the site. python.org Depending upon your web host install, you can easily install python as a package using the control panel. You may need to search you hosts help pages or chat with them. It could be installed already! Who knows? – closetnoc Apr 16 '18 at 15:22
0

You can identify (good) bots by their user agent in PHP with HTTP_USER_AGENT.

Here is an example:

if(strstr(strtolower($_SERVER['HTTP_USER_AGENT']), "googlebot"))
{
    // what to do
}

Source: https://stackoverflow.com/questions/677419/how-to-detect-search-engine-bots-with-php

Here is a directory of bot names: http://www.searchenginedictionary.com/spider-names.shtml

You can also detect bots in your log files on your server. Good bots will have their user agent listed in your logs. A log viewer application can sort the IP addresses of user agents chronologically so that you can see how many visits each one made. Or, after downloading the log file, you can search the log for the bot name and based on how many results there are will be how many times that bot crawled your site.

|improve this answer
  • Agent names can easily be faked and a poor indicator of a good bot versus a bad bot. In fact, it is the worst thing you can do. It is so unreliable that it is NOT an indicator at all! Good bots and bad bots behave differently and can be separated rather easily. However, this takes some real expertise. I should know. It was my business for a long time. The OP should be using Fail2Ban or other product and not try and roll their own. I list some of the characters of a bad bot in this answer. webmasters.stackexchange.com/questions/67221/… – closetnoc Apr 16 '18 at 6:23
  • Thanks. But none of the bots my third-party tracker recorded is listed in the list of searchenginedictionary.com. That list doesn't seem to be very up-to-date. – Ian Y. Apr 16 '18 at 7:35
  • If you're getting bots that aren't from major crawlers, go through your logs and identify which bots are hitting you the most. Search their IP addresses. If those IP addresses aren't registered to major companies then you may need to block them. – Michael d Apr 16 '18 at 7:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.