One of the sites I maintain has seen Googlebot use 23gb of bandwidth this month.

This seems utterly mental to me, what can I do about it?

I have reduced the crawl rate on webmaster tools, and edited the robots.txt, submitted a sitemap.

A serious spike like this feels like a hack, could Googlebot be spoofed?


It's very simple to spoof a User-Agent header and pretend to be a Google Bot. But much harder to fake the I.P. from which the request is coming.

Check that the I.Ps making these request are owned by Google.

| improve this answer | |
  • should that prove to be the case, what is the best course of action? – Mild Fuzz Oct 27 '11 at 17:21
  • It seems that you've done the right thing by changing crawl speed the settings - you can wait and see if that works – Andy Smith Oct 28 '11 at 8:42

Googlebot does not follow crawl-rate (as reported by Google's webmaster tools). But it's good to set anyway for other bots. Check for potential endless loops on pages (such as search results or article listings with different ways of sorting/filtering, etc. and apply a noindex or nofollow to them and/or a link rel=canonical page for them so Google only sees them once.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.