I'm analyzing web server logs both in Apache and IIS log formats. I want to find the evidence for automatic browsing, like web robots, spiders, bots, etc. I used python robot-detection 0.2.8 for detecting robots in my log files, but I know there may be other robots (automatic programs) which have traversed through the web site but robot-detection can not identify.
So I want to ask:
- Are there any specific clues that can be found in log files that human users do not leave but automated software would?
- Do they follow a specific navigation pattern?
- I saw some requests for
favicon.ico- does this implicate that it is a automatic browsing?.