I am requested to analyze the traffic log of a site. In particular I have to identify the crawlers starting from the collected user agent
values.
I know there are 'trap' links that you can use to distinguish the crawlers from the human beings. Now I would only analyze the user agent
values.
Now the question. Is there a public catalogue or a library of web crawlers?
Edit
Here is the second question. There are also a lot of empty user-agent in my traffic records. Is an empty user-agent header related to a crawler or to an authomatic process?