I'm seeing in my access log a number of request very suspicious:


They part from a known resource (in the above example /images/disrupt.jpg).

All comming from same IP. Requests varies from 1/sec to 10/sec, seems somewhat random.
It's obviously they are trying to find something and seems they are using a script.

How do I block this kind of behaviour? I though of blocking the IP request, at least for a given time. Keeping in mind that:

  • Request intervals seems legitimate (at least I think so).
  • I don't want to end blocking a search engine bot, which may find 404 urls too (and that's a different problem, I know). ¿Do they use always same IP?
  • The access log will tell you the user agent. Search engines should be obvious, e.g. "Googlebot". If it says something like "Mozilla/5.0" at the front then it's probably a human. – DisgruntledGoat Nov 18 '11 at 12:40

Do they use always same IP?

No, search engines can be expected to use a variety of IP addresses - but they do always use the same autonomous system (and all the major search engines have their own AS).

If you have the IP address, you can go to ARIN and use the "WHOIS Search" at the upper right-hand corner of the page to look up the autonomous system associated with the IP address.

| improve this answer | |

Finally I found who was the responsible, it was a javascript that tried to load the resources in real time as somebody write an article.
As the user was typing the url of an image, the script tried to load it even if the path was not complete, hence that 404 logs.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.