Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I'm seeing in my access log a number of request very suspicious:

/i
/im
/imaa
/imag
/image
/images
/images/d 
/images/di
/images/dis

They part from a known resource (in the above example /images/disrupt.jpg).

All comming from same IP. Requests varies from 1/sec to 10/sec, seems somewhat random.
It's obviously they are trying to find something and seems they are using a script.

How do I block this kind of behaviour? I though of blocking the IP request, at least for a given time. Keeping in mind that:

  • Request intervals seems legitimate (at least I think so).
  • I don't want to end blocking a search engine bot, which may find 404 urls too (and that's a different problem, I know). ¿Do they use always same IP?
share|improve this question
    
The access log will tell you the user agent. Search engines should be obvious, e.g. "Googlebot". If it says something like "Mozilla/5.0" at the front then it's probably a human. –  DisgruntledGoat Nov 18 '11 at 12:40
add comment

2 Answers 2

Do they use always same IP?

No, search engines can be expected to use a variety of IP addresses - but they do always use the same autonomous system (and all the major search engines have their own AS).

If you have the IP address, you can go to ARIN and use the "WHOIS Search" at the upper right-hand corner of the page to look up the autonomous system associated with the IP address.

share|improve this answer
add comment
up vote 0 down vote accepted

Finally I found who was the responsible, it was a javascript that tried to load the resources in real time as somebody write an article.
As the user was typing the url of an image, the script tried to load it even if the path was not complete, hence that 404 logs.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.