1

The last week I have about 500.000 hits/day from googlebot in one of my websites. As a result, the server has serious issues to handle the rest of the requests.

What I have done

  1. Verified that it is a googlebot

All the IPs are in the format of 66.249.64.xxx. I followed what Google suggests and verified that it is official.

host 66.249.64.155
155.64.249.66.in-addr.arpa domain name pointer crawl-66-249-64-155.googlebot.com

host crawl-66-249-64-155.googlebot.com
crawl-66-249-64-155.googlebot.com has address 66.249.64.155
  1. Reduce the crawl rate in the Google Search Console

Changed the "Limit Google's maximum crawl rate" to 

0.01 requests per second
100 seconds between requests
  1. Add a block on robots.txt
User-agent: Googlebot 
Disallow: /
  1. Add an .htaccess rule
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^googlebot
RewriteRule ^(.*)$ http://google.com/

I also found a page in Google that shows the Crawl Stats for the last 90 days. As I can see, it doesn't have any recent activity. When the last was on February (again with a high number of hits)

enter image description here

My last finding is the pages that crawl are from the shop part with all the different filters in the parameters like categories, tags and sorting. I am clearly out of ideas and Google Search helped as much as it could.

  1. Exclude parameters from the Search Console tool https://www.google.com/webmasters/tools/crawl-url-parameters

I did this today, so it might need a couple of days for Google to apply it. I excluded the parameters, min_price, max_price, filter_size, filter_color, filter_brand, filter_type and orderby.

This is an example of the URL that GoogleBot hits

https://www.example.com/product-category/woman/?filter_size=36&query_type_size=or&min_price=0&max_price=30

Improve this question
8
  • 1
    This is very unusual behavior for Googlebot. Googlebot is normally a very well behaved bot that is very careful not to put too much load on the the sites that it crawls.
    – Stephen Ostermiller ♦
    Apr 5, 2020 at 12:20
  • 1
    What is your site, and how many pages are on your site?
    – Stephen Ostermiller ♦
    Apr 5, 2020 at 12:20
  • Warning: your Disallow: / rule for Googlebot will remove most of your site from Google search within a couple weeks.
    – Stephen Ostermiller ♦
    Apr 5, 2020 at 12:21
  • The crawl stats graph from Google search engine console may take a few days to update. GSEC often doesn't have the must up-to-date information in it.
    – Stephen Ostermiller ♦
    Apr 5, 2020 at 12:23
  • I know about the robots.txt that will have a negative impact in the SEO, but I had to take an action until there is a more proper solution. The website in an eshop with about 250 products and 25 categories. Plus some tags. All the crawled urls are from the /shop page where it tries with all the possible parameters of sorting and categories.
    – Tasos
    Apr 5, 2020 at 13:44

1 Answer 1

Reset to default
1

I tend to think it's probably not Googlebot, it's quite easy to spoof the User Agent string, and with a bit more effort to report a false IP address.

Regardless, you're basically experiencing a Denial of Service attack so a service designed to prevent that (query 'edge server' on the search engine of your choice), or anything else that lightens the load on Apache, would probably help.

I almost always use Nginx as a reverse proxy in front of Apache for a variety of reasons but cache and efficiency are a big part of it.

There is a way to set a noindex HTTP header in Apache but if it's ignoring robots.txt then I doubt that would help.

Improve this answer
2
  • 1
    The steps outlined in the question mean that it isn't just user-agent spoofing. The bot is using an IP address owned by Google. See Verifying Googlebot - Search Console Help where they talk about using this method to know if hits are actually from Googlebot or from a spoofed user agent. I upvoted this answer because the rest of the content is good.
    – Stephen Ostermiller ♦
    May 9, 2020 at 12:49
  • @StephenOstermiller I agree it does appear to be from Google but sometimes when I look through my access logs, and check an I.P. address, I find it's completely non-existent. Because most people would never want to block GoogleBot, if I wanted to harass someone and bog down their server, I might fake a GoogleBot UserAgent string and use a tool like hexinject.sourceforge.net to also spoof a Google I.P. address...but I realize that is probably a stretch (and Google does have hiccups now and again) but that was my logic.
    – adam-asdf
    May 9, 2020 at 22:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.