I am dealing with very heavy load from Googlebot, I have hundreds of sites running on multiple IPs all on the same server and googlebot is constantly crawling these sites causing a lot of availability issues to those sites.

I've checked the IPs and they seem to be legit (66.249.66.XX) so it doesn't look like its other bots spoofing, but i see weird things like google using negative numbers for paging, for example ?page=-274. I've even had to add code to check if the value is < 0 to use 1 instead to avoid problems.

Any idea why is google doing things like that and how to prevent it?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.