In blocking a lot of scrapping activity, I found a puzzling situation - that my site is accessed about 35,000 times per day by around 30,000 different IPs. I started blocking these IPs with a script, which blocked a portion of this traffic, but there would always be more IPs. The IPs are mostly from Asian ISPs (residential broadband lines), but there are requests from IPs from many countries. Most of the requests are for the homepage, and about 250 requests per day for six sub-pages (the same ones). They don't download any scripts, images etc. and have no referer.

I've been able to block these requests as they all use slight variations of a Chrome user agent with the wrong version of Webkit for that Chrome version.

But I am wondering why would anyone do this? Could it potentially have an affect on rankings?

My deduction is that the requests come from a bot net, a collection of infected computers that are used to proxy web requests to get around rate limiting. They are typically used for web scrapping and can be bought from services like ProxyRack.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.