1

We are planning to start program something similar to youtube where users get paid as per views.

But problem we are facing is that people will game the system and send traffic using botnets to gain undue advantage.

I know there might be no 100% legit way to uncount such traffic but what is the best way to filter the traffic?

  • This is too broad of a topic- I could easily write a book! However, you can help us narrow it down. Are you referring to Google Analytics? Are you using a log file analyzer? – closetnoc Jun 19 '15 at 17:47
3

Since you use tags that start with web, I assume you are building a system that requires HTML to be downloaded in order for a paid impression to work.

What you need to do is learn robot behaviour by looking at the server access log files. On a server with apache installed, the file is typically named access_log. If your server is used frequently, you will see hundreds, if not thousands of lines in it. Each line contains the IP address of the remote device that's connected to your server as well as the resource it requested along with the date and time.

Generally, there's at least one second between the time a person goes from one page to the next on a website unless of course the website is a poorly designed high-speed guessing game with no instructions, and even then, the speed might never be faster than one second due to initial connection latency.

If you see the same IP address listed along with the exact same date and time for the next 20+ lines straight, then its clearly a robot trying to attack your system (maybe trying to produce a denial of service attack).

Another way to check if its a robot is to look for the files being requested, especially in the error logs. If you constantly see a similar pattern of files over and over again, especially if none of them exist, then it means one or more systems may be trying to break into your server by assuming you have content management systems installed such as Wordpress.

Also, some robots may be requesting files with screwed up names, or may be falsely identifying themselves. Any line containing source code is likely a line coming from a robot. For example, code containing this: (:;){}

Because I'm unsure the name of the server (apache? nginx?) is serving the content on your system, I'm unable to tell you which column in the log files belong to the IP address or which belong to the file being requested etc., but when you look at them, especially after you make a request to the server, the patterns may become easier to discover.

| improve this answer | |
2

A few more methods of bot traffic detecting:

  1. Verification of user agent (it is a client application that uses a particular network protocol).
  2. Looking for a highly specific match to something like a malware signature or specific executable or C&C connection address.
  3. Examining such behavioral parameters as a depth of view, the duration of visit, engagement and some other parameters.
  4. Program solutions. There are two known by me solutions to filter traffic and to examine its quality - Google Analytics and Maxymizely.com. Use GA to filter hits from known botnets and referrers. To filter bot and spider traffic from Google Analytics, go to your Admin settings -> View Settings -> Bot Filtering with a checkbox that reads ‘Exclude all hits from known bots and spiders’. The second solution Maxymizely.com allows to analyze traffic quality considering three dimensions - activity, engagement and monetization of traffic. This tool allows to choose the significant sub-parameters for every dimension and to assign required weight to them. Also, there is a handy opportunity to visualize this stuff in a view of 3D map.
| improve this answer | |
1

Find out how many videos viewed by normal users per hour.

Let's assume It is 10 videos viewed by most of visitors per hour.

Store & use the # of videos viewed using session + cookie + IP address.

If the number of videos viewed are higher than 10 for any session, then use simple captcha to avoid the bots.

If the captcha verification is failed ( continuously or many times) add these IP addresses in black list and treat them in a special way. I.e introduce more captcha in a incremental way.

You can find out the difference between bots and users easily using the combination of IP Address + Cookie + Session variables.

| improve this answer | |
-2

Get a list of known referral spam URL's and filter them through Analytics.

https://megalytic.com/blog/how-to-filter-out-fake-referrals-and-other-google-analytics-spam

| improve this answer | |
  • While the link may help answer the question, a link only answer is not high quality. A better answer would contain the full solution in the answer and use links for reference and sources. A user shouldn't have to click on any links to be able to implement your advice. – Stephen Ostermiller Jun 19 '15 at 22:56
  • The answer is there, you nut. What do you want? A screenshot tutorial? "The user" should be able to find the "filter" button in analytics. – Ethan Jun 21 '15 at 4:55
  • Paraphrasing the major points from the tutorial here would make this a much better answer. Copying or taking a screen shot would not be appropriate as it would violate copyright laws. – Stephen Ostermiller Jun 21 '15 at 7:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.