1

My website is currently getting hit with spamming bots (Example: 66.249.73.*) that are causing high CPU usage. Is it common for Google/Bing to crawl a website 10/sec?

I have done reverse look ups on the IPs and they appear to be valid crawlers using https://support.google.com/webmasters/answer/80553?hl=en.

Because I have blocked some IP's, Google/Bing Search Console are reporting errors and hurting my index.

This has been happening this month (April). Is this a referral attack? Is it possible someone is spoofing the IPs? Is there something I can do to limit the amount of crawls?

I am currently in the process of creating a server-side rendering for crawlers, but this is a tedious task for something that just started happening randomly.

  • If your CPU heavily on legit, or even rogue bots, it's time to upgrade your hosting. Your website hosting should be able to handle users, legit bots, and even 3rd bots. – Simon Hayter Apr 30 at 22:27
  • How many pages has your website ? If you've changed something on your website which concern a lot of website, Googlebot might feel hhat it requires to crawl the whole website very quickly. – Quentin May 1 at 18:22
3

Google uses many IP ranges. From the one you posted, any IP from them in the range 66.249.64.0 - 66.249.95.255 and identifying itself as Googlebot should be a legit bot.

There are many reasons for the increase in crawl rate, maybe some of your content is viral, or their bot wants to refreshen the data from your site sooner. It usually IS a good thing.

I would NEVER block a Google IP range, unless you don't want visitors to reach your site. What you CAN do if it's hammering your resources is to specify a crawl-delay for other search bots in Robots.txt.

Google does not support the Crawl-delay directive. However, Google does support defining a crawl rate in Google Search Console.

  • Thanks for the response. This info was helpful. – Matt Apr 30 at 21:55
1

If the IP addresses from where the requests are originating are identified as that of Google/Bing then you should not block them as this will impact your SEO. Instead of blocking them you should adjust the crawl rate in their respective Webmaster tools

Both Google and Bing offer the ability to adjust crawl rate with good flexibility.

Change Googlebot crawl rate

Bing Crawl Control

In the case of Yandex crawling your site you can add a Crawl-delay directive to slow down the crawl rate for yandex

If you think there are spam bots that are affecting your servers (which can be determined by observing the server logs) consider using a Web Application Firewall that will block suspicious IP Addresses. Cloudflare has the ability to allow known bots and block suspicious bots based on the threat score which it calculates. Additionally you can also block certain User-agents from crawling the website

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.