1

The content on my site is largely PDF. I am using awstats to gather stats, knowing full well the limitations of a server log based analyzer, and that a lot of downloads are generated by bots. Please note that I really don't want to use Google Analytics or Matomo because I really dislike their interfaces, and really like the interface awstats has. This is also true for my clients.

Being the roll-your-own re-invent-the-wheel kinda guy I am, I am looking for ways to detect if a "user" is actually a bot. Maybe looking for certain patterns in the logs, or a way to detect activity when viewing the PDF in the browser.

3 Answers 3

3

There is no silver bullet, especially with AI, but there are a few thoughts no one above has mentioned -

  1. Implementing some kind of Javascript required to complete the URL's or similar can frustrate a lot of bots - as many won't know how to process it. If you think about it, that is a significant part of what Google Analytics relies on.

  2. Using a CAPTCHA or equivalent?

  3. There are no doubt lots of places you can get lists of IP addresses used by bots - one such list would be cleantalk.org.

  4. Can you narrow reference down to a geographically located area, and use a geolocator database?

  5. It won't "fix" the problem, but if bots are heavily spidering a site, something like fail2ban may help detect IP's

  6. Can you have (a possibly trivial) login to the site? Maybe combined with a cookie/session tracking?

2

The most common ways to identify bots via server logs would be IP address analysis to figure out if the IPs are coming from sources likely to be bot farms. There is an inherent inaccuracy because sometimes traffic could originate from residential sources but still be bot traffic. There is also IP forging which would increase the inaccuracy of looking only at the IPs.

You could weed out bots by looking at the user-agent and cross referencing known bot traffic. This might not tell you a whole bunch because it can easily be changed on the scripting side.

Behavior analysis and looking at how the "user" is floating through your site and how quickly they are accessing links/PDFs/etc.

There are probably a bunch more things you could do but it will be pretty in depth where you will probably need multiple datasets to augment your server logs.

It would probably be more beneficial to you to implement technology to identify and prevent bots if it is that much of a concern (application management software, captchas, etc.).

1

There are heuristics you could use, such as

  • User agent is a familiar browser, and the version is recent.
  • Correct referrers. I've seen bots going through my whole website with referrers being existing pages, but the referrer of each page not linking to it. Some bots make requests just to advertise the sites in the referrer.
  • Reasonable browsing rate. It takes humans time to read a page, find a link, and click it. Bots often run much faster through the website.

There are two problems that can't be avoided. The first is bot creators have become proficient in subverting protection mechanisms. There's at least one company that provides content framing as a service (display any page their customer desires in the customer's web site), and their proxies are good in presenting themselves as a real user. The second is the underlying protocols were not designed to make the users from bots. Combined with grid and dubious 'ideologies' a lot of people think that any all web sites are free for all resources. They may hotlink images, frame PDFs to present the content as their own, archive web sites for historical research, etc. Which is why I think the Internet protocols are defective by design.

10
  • You have some good points byut Some of what you say is a stretch - User agents are trivially and commonly faked, content framing is not a bot acrivity, hotlinking is trivial to block and again not not activity, and just internet protocols are not deffective by design - however theycan not be all things to all people for all time.
    – davidgo
    Feb 29 at 17:12
  • @davidgo SMTP is an excellent example of defective my design. It was not designed to become obsolete, but nobody ever added mechanisms to prevent lasting problems, such as spam. A trivial example would be virtual stamps. Pay .1c to send an email, and receive .1c to receive an email, and though normal usage would cost practically nothing, but spam would become expensive. ISP would rather charge both the spammers and the recipients. A proxy designed for content theft is an automaton, a bot.
    – Uri Raz
    Feb 29 at 18:07
  • This is untrue. Email is one of the earliest protocols (RFC821, 1982)- well before it was envisaged that non-academics would be using it. There is absolutely nothing stopping "stamps" being added to the protocol - except that it won't (and cant) be adopted for a raft of reasons - its not like this idea is even new. While it can be used as part of an MITM attack, a proxy is neither designed for theft nor can it be a bot. This is a bit like saying knives are defective by design because people can hurt themselves and others with them.
    – davidgo
    Feb 29 at 20:29
  • And, as someone that has been in the ISP industry since the early days, I can assure you that neither me nor my colleagues want to charge recipients for receiving email - although responsible players have played with charging senders a small fee as your stamp suggestion. You seem to be confusing SMS in America with email.
    – davidgo
    Feb 29 at 20:33
  • @davidgo SMTP has been extended many times since 1982. Stamps are easy - the SMTP server can drop the connection if the user fails to authenticate, e.g. POP before SMTP with TLS. It doesn't happen because ISPs can charge a fee, and keep it in their pockets. As for proxies, the method is the customer adds a URL to the proxy into the HTML file, the proxy checks which page it should fetch, request it as if its an innocent user, strip any headers that limit framing, and pass it to the browser. Voila! The customer has content displayed from another site, which pays for the bandwidth,
    – Uri Raz
    Feb 29 at 23:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.