2

Certain visitors on my site request more than one webpage per second on average. For example, I see the following statistics:

ip         visits  timestsamps visits per second
1.2.3.4    500     500         1
1.0.0.1    121     60          2.016667
2.2.2.2    100     32          3.125

where

  • "ip" (here made up) is the visitor's client ip
  • "visit" is the total number of visits during a period of time
  • "timestamps" is the number of different timestamps (in whole seconds)
  • "visits per second" is the average number of visits per second

If the average number of visits per second were the only information I had, would it be possible to say that some of those visitors must be bots/crawlers/etc. ?

What is the highest number of average visits per second that's plausible for human site visitors?


Notes.

Visits here only count requests to HTML pages, not requests for resources embedded in HTML pages, such as images, style sheets, and so on. I understand that calling an image-rich HTML page would lead to many requests within a few microseconds of each other, so I'm excluding requests for those resources from this statistic.

Also, the visits aren't necessarily consecutive. While it is unlikely that a human visitor would browse 500 webpages one after another in exactly 500 seconds, a visitor could return 500 times over the course of a year leaving 500 different timestamps. But once the visits per second go above 1, that means that this visitor has requested one or more pages within one second of each other. And this is something that human visitors don't necessarily do, at least not at a consistent high speed. Calling 100 pages at three pages per second seems "inhuman" to me. Or is that still common human browsing behavior?

  • Google.com show capcha on some networks, cause large network with few ip looks like bot. You may use this on your site, but avoid blocking real good bots, like googlebot - either by ip or useragent. – LeonidMew Jan 7 at 10:46
  • @LeonidMew I don't want to block bots, just recognize them in my logfiles. – user95497 Jan 7 at 15:10
  • Then I suggest add few rules, to you bot detection algorithm, like 5 requests in a second, 50 in a minute, 300 in 30 minutes. Tune it, and if any of rules hit - this is a bot. Note, that also detect slow bots, as they do many requests anyway, more than human can. – LeonidMew Jan 7 at 15:34
0

You need to be careful when using this approach. It is common for a large number of users to appear to come from the same IP address. That means that it is plausible that certain IP address will have lots of hits representing many users.

There are two main situations in which this happens:

  1. Proxy servers
  2. Corporate gateways

Some popular proxy servers may host a large number of visitors. For example when AOL was popular, all of its users used proxy servers. All AOL visitors would appear to come from a few IP addresses. When we were trying to detect bots by IP address activity we would have to make exceptions for AOL's proxy server IP addresses. Of course AOL isn't big today anymore, but you may find other proxy servers that are used by a number of your visitors.

It is also common for a large corporation to use a single IP address from which all of their employees appear to access the internet. One IP address may represent the activity of thousands of employees of a single corporation.

I would not recommend blocking high activity IP addresses without some sort of further validation of bot activity. Other things to check when an IP address has lots of hits:

  • Is the user agent always the same, or does it seem to represent a set of users each with their own user agent?
  • Do the hits seem regular or sporadic? Bots ore much more likely to be regular while users are likely to be somewhat irregular.
  • Do the hits support cookies?
  • Do the hits represent full page views? Do you see appropriate hits for resources such as CSS and images?
  • Does the browser appear to support JavaScript? Do you see all the hits for the AJAX and websockets that your site might be using?
  • Do the hits include any honey pot URLs that you are hiding on your site? You can include links visible in your source code but hidden from users for the purpose of discerning bots.

Also keep in mind that just because it is a bot, you still may want to allow it. If you block Googlbot or other search engine robots, your site will get de-indexed from the search engines.

  • Thank you, the first part of your answer is helpful, but I don't think it applies in my case. My website has an average of 200 pageviews per day, so it is unlikely that suddenly a group of people behind the same proxy visit it at the same time. Also, my question asks: "If the average number of visits per second were the only information I had..." I want to understand the speed of page views, nothing else. I want to use the understanding of this single factor in conjunction with others, some of which you have mentioned, but I want to understand user behavior first. – user95497 Jan 7 at 15:18
  • 1
    The rate of users' page views is going to vary dramatically between different types of sites. I visit image sites where you just keep clicking to the the next image. On those types of sites, it would be very common to visit many pages in a row spending a second or less on each page. You may find users also navigate through some pages quickly but spend more time once they get to meaty content. Users may also click many links on your site to open the contents in many new tabs. – Stephen Ostermiller Jan 7 at 15:36
  • 1
    So to answer your question: I don't think it is possible to use only the rate of hits to determine whether or not an IP address represents a bot. I'd certainly be suspicious of the behavior you describe, but I'd look for more information before making any decisions such as banning IP addresses. – Stephen Ostermiller Jan 7 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy