Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I just did some log analysis of Q1 for one of my client's sites.

There were about 30,000 sessions created for this site for each of the five separate IP addresses starting with 79.171.81.xxx. 150,000 sessions in three months on this site represents about 30% of the total traffic.

The visits are representing themselves with a consistent User-Agent string of Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)

Does this IP range belong to a known (beneficial) spider of some kind? Or is it more likely to be something malicious? I could take many steps including:

  • ignore the volume and let it access the site as much as it likes
  • turn off logging for this IP range
  • completely block it from accessing the site
  • anything else!
  • email the abuse@ address listed for the IP address and ask for clarification (or is this just asking for trouble?)

Googling for the IP addresses just gives me lots of IP lookup results.

So, I submit myself to the wider knowledge and wisdom of the crowd: what's going on here and what should I do about it (if anything)?

share|improve this question

closed as too localized by Su', Simon Hayter, John Conde Apr 11 '13 at 20:43

This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center.If this question can be reworded to fit the rules in the help center, please edit the question.

up vote 1 down vote accepted

First, inform your client and ask what they would like to do - it's still their site - and it could be a major customer's traffic or another source you're not privvy to.

If client knows nothing of the traffic source, suggest blocking the IP range & monitor load. Notify the client as soon as the block has been implemented.

  • If the traffic jumps to another IP, you know something's up.
  • If your client's sales team gets a phone call from irate customer(s), then you also know the source and you can now prioritize that traffic.
share|improve this answer

Any properly run robot would have information in the User-Agent that identifies who is running it.

That is not to say that this is necessarily being run for less than legit reasons, but the absence of identifying information in the User-Agent string certainly doesn't speak well for their etiquette. I'd block them for that alone.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.