1

This morning in logs I noticed a user got rate limited by our throttler 3 times within 1 second (429 too many requests error) at 3:30:11 AM EST: Mozilla/5.0 (Linux; Android 6.0; GCE x86 phone Build/MRA59G.MZC35) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2679.0 Safari/537.36 at IP 104.196.83.207. I've only seen Google on this IP once before, so I'm unsure what it's purpose is.

Upon looking it up it seems it's a Googlebot indeed. Noted below by Analog, its coming from Google Cloud. The last paths it was trying before being limited for 30 seconds were images. We have never had problems with Google speeding the site until last night. Our throttler is set to trigger a 30 sec 429 error if a user hits 10 requests in a 5 second window (session var based). 3 logs imply they hit this limit 3 times within 1 second (30 total requests per sec).

Right now, the Search Console setting says it's on the "let Google decide" -- clicking the radio to change rate puts it square in the middle at "1 request every 10 seconds". On its maximum setting it allows 2 requests a second (ie throttle zone).

I will adjust the crawl rate manually to 1 every 10 seconds, but in the meantime why is this bot crawling so fast, and why is it on android mobile crawling only images?

Improve this question
1
  • 1
    Google will not access your site more than one request per 2 seconds by default unless you tell them otherwise. Anything else are likely hosted sites or the google apps thing which have both have been used to abuse sites. If you see 104... you know it is junk!
    – closetnoc
    Commented Mar 15, 2016 at 15:44

1 Answer 1

Reset to default
3

It is my understand that all google bot crawler IP addresses can be identified by the RDNS which is set to something along the lines of "crawl-(IP ADDRESS).googlebot.com".

The useragent you mentioned is also not used by the googlebot. Google lists the useragent strings of all their bots here.

The IP address you posted is part of the netblock used by Google Cloud customers as seen in the ASN comments

NetRange:       104.196.0.0 - 104.199.255.255
CIDR:           104.196.0.0/14
NetName:        GOOGLE-CLOUD
NetHandle:      NET-104-196-0-0-1
Parent:         NET104 (NET-104-0-0-0-0)
NetType:        Direct Allocation
OriginAS:       AS15169
Organization:   Google Inc. (GOOGL-2)
RegDate:        2014-08-27
Updated:        2015-09-21
Comment:        ** The IP addresses under this netblock are in use by Google Cloud customers ** 
Comment:        
Comment:        Direct all copyright and legal complaints to 
Comment:        https://support.google.com/legal/go/report
Comment:        
Comment:        Direct all spam and abuse complaints to 
Comment:        https://support.google.com/code/go/gce_abuse_report
Comment:        
Comment:        For fastest response, use the relevant forms above.
Comment:        
Comment:        Complaints can also be sent to the GC Abuse desk 
Comment:        ([email protected]) 
Comment:        but may have longer turnaround times.
Comment:        
Comment:        Complaints sent to any other POC will be ignored.
Improve this answer
4
  • 2
    Ahhhh thank you, i didn't even think of their cloud offerings. I should have spent a sec rather than just db-ip.com it :) Prob yet another image scraper then -- some person is obsessed with making mis-spelled domain scraper blogs of home hearth/heating/mechanical systems images, I dunno why.
    – dhaupin
    Commented Mar 15, 2016 at 15:10
  • 2
    Yep would seem like someone trying to scrap your content/images for their own use. You could report the IP to google for excessive web crawling - support.google.com/code/contact/compute_engine_report?rd=1 Don't forget to reset your google crawl settings in webmaster tools if you have changed them! :)
    – Analog
    Commented Mar 15, 2016 at 15:20
  • 1
    Perfect answer!!
    – closetnoc
    Commented Mar 15, 2016 at 15:42
  • @Analog Thanks for the link, I submitted some logs to Google about this IP and reset that crawl rate back to "let G decide".
    – dhaupin
    Commented Mar 15, 2016 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.