Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I'm analyzing web server logs both in Apache and IIS log formats. I want to find the evidence for automatic browsing, like web robots, spiders, bots, etc. I used python robot-detection 0.2.8 for detecting robots in my log files, but I know there may be other robots (automatic programs) which have traversed through the web site but robot-detection can not identify.

So I want to ask:

  1. Are there any specific clues that can be found in log files that human users do not leave but automated software would?
  2. Do they follow a specific navigation pattern?
  3. I saw some requests for favicon.ico - does this implicate that it is a automatic browsing?.

I found this article and this question with some valuable points.

share|improve this question
What do you wish to achieve ? –  bybe Jun 30 '13 at 12:55
@bybe : I want to find evidences of automatic web browsing(not done by a human) for a data mining research purpose? –  Nilani Algiriyage Jun 30 '13 at 13:03
There are many bots/scapers that do not leave a footprint though, they do not use a bot user agent but use browser agent also they may only scrape data from the home page which would leave no digital footprint to estiblish if these are a bot or not, also there are software scrapers that use random user agents based on Chrome, Firefox, IE etc that will scrape a few pages for several reasons, again it'll be very hard to unlikely you will be able to detect these. –  bybe Jun 30 '13 at 13:38
@bybe : Ok thanks bybe! Hope I could figure out some ways to detect them up to some extent! –  Nilani Algiriyage Jun 30 '13 at 13:42

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.