I just did some log analysis of Q1 for one of my client's sites.
There were about 30,000 sessions created for this site for each of the five separate IP addresses starting with 79.171.81.xxx. 150,000 sessions in three months on this site represents about 30% of the total traffic.
The visits are representing themselves with a consistent User-Agent string of
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
Does this IP range belong to a known (beneficial) spider of some kind? Or is it more likely to be something malicious? I could take many steps including:
- ignore the volume and let it access the site as much as it likes
- turn off logging for this IP range
- completely block it from accessing the site
- anything else!
- email the abuse@ address listed for the IP address and ask for clarification (or is this just asking for trouble?)
Googling for the IP addresses just gives me lots of IP lookup results.
So, I submit myself to the wider knowledge and wisdom of the crowd: what's going on here and what should I do about it (if anything)?