Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 1 down vote favorite

One of the sites I maintain has seen googlebot use 23gb of bandwidth this month.

This seems utterly mental to me, what can I do about it?

I have reduced the crawl rate on webmaster tools, and edited the robots.txt, submitted a sitemap.

A serious spike like this feels like a hack, could gbot be spoofed?

share|improve this question

migrated from serverfault.com Dec 1 '11 at 21:59

2 Answers

up vote 8 down vote

It's very simple to spoof a User-Agent header and pretend to be a Google Bot. But much harder to fake the I.P. from which the request is coming.

Check that the I.Ps making these request are owned by Google.

share|improve this answer
should that prove to be the case, what is the best course of action? – Mild Fuzz Oct 27 '11 at 17:21
It seems that you've done the right thing by changing crawl speed the settings - you can wait and see if that works – Andy Smith Oct 28 '11 at 8:42
up vote 3 down vote

Googlebot does not follow crawl-rate (as reported by Google's webmaster tools). But it's good to set anyway for other bots. Check for potential endless loops on pages (such as search results or article listings with different ways of sorting/filtering, etc. and apply a noindex or nofollow to them and/or a link rel=canonical page for them so Google only sees them once.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.