Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Let me show you what I mean:

IP               Pages   Hits    Bandwidth
85.xx.xx.xxx     236     236     735.00 KB
195.xx.xxx.xx    164     164     533.74 KB
95.xxx.xxx.xxx   90      90      293.47 KB

It's very clear that these person are crawling my site with bots. There's no way that you could visit my site and use <1MB bandwidth. You might say that there's the possibility that they could be browsing the site using some browser or plug-in that does not download images, js/css files, etc., but the simple fact of the matter is that there are not 90-236 pages that are linked from the home page (outside of WP files), even if you visited every page twice.

I could understand if these people were crawling the site for pictures, but once again, the bandwidth indicates that this isn't what is happening. Why, then, would they crawl the site to simply view the HTML/txt/js/etc. files?

The only thing that I can come up with is that they are scanning for outdated versions of WordPress, SQL injection vulnerabilities, etc., which makes me inclined to outright ban the IPs, but I'm curious, is it possible that this person is a legitimate user, or at the very least, not intending to be harmful?

share|improve this question
    
Have you done a reverse DNS lookup to find out what domains those IP addresses belong to? Do they belong to search engine companies or are they ISPs/domestic users? –  Nick Jun 21 '11 at 9:11
add comment

2 Answers

up vote 1 down vote accepted

I've noticed several IPs spidering my company's main web sites in entirety over the past 12 hours - it's nothing unusual. I suspect that as at least one of them appears to be a regular residential cable broadband connection here in the UK, they may well be people performing SEO-related indexing in order to gauge keyword effectiveness before trying to game a keyword in Google (and our sites are just one of many hits from their machine as part of that).

Either that or they're looking for vulns!

share|improve this answer
add comment

Some bots simply crawl just sites to index content. They're trying to do something like Google crawler and they do try it on your site. Or they do download it with

file_get_contents("yoursite");

and similar methods which do not retreive images/css/js

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.