1

I recently saw a significant googlebot crawl on my website. We're talking 10-20 requests per second. It looked like googlebot, and seemed to be legit googlebot IPs, but the requests had query params that were encoded for korean and chinese characters which appeared to be translatable as spam.

I was able to mitigate by updating my robots.txt to stop crawling.

Question, how does googlebot get into this state? How was abuser able to create these spam queries in such a way that amplified googlebot requests to my site? Is this a commonly exploitable flaw in the crawler? Seems like it can be easily done again as a DDoS on a different site or route.

1 Answer 1

3

I guess Google thought you were a big host with all these new pages and could handle the speed since you have 1000s of pages google did not know about? Under other circumstances that would be a good thing.

You can tell googlebot how fast to crawl your site in/at

https://www.google.com/webmasters/tools/settings

With documentation here: https://support.google.com/webmasters/answer/48620?hl=en.

Google has sophisticated algorithms to determine the optimal crawl speed for a site. Our goal is to crawl as many pages from your site as we can on each visit without overwhelming your server's bandwidth.

If Google is making too many requests per second to your site and slowing down your server, you can limit how fast Google crawls your site.

Alternatively, A 429 status code, "Too Many Requests," can be sent when the site is under stress. Kills two problems with one solution.

For Apache level DDos a mod_evasive is useful

I understand there is a fork https://serverfault.com/a/877762/89279 or apache mod_evasive that sends a 429 instead of the 403 during an DDos.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.