Does Googlebot, when crawling pages which display maps using the Javascript API, cause a billable API usage on each page request?
2 Answers
Yes. At the end of August, a 20-fold increase in the Googlebot crawl rate pushed my site into the paying band for the Maps API. Three days of being monstered cost me €174.
Fish in a barrel?
An investigation into the Google Robots.txt file...
https://www.google.com/robots.txt
Allow: /maps?*output=classic*
Allow: /maps?*file=
Allow: /maps/api/js?
Allow: /maps/d/
Has Google's own crawler pushed up the maps views on your website? It looks like it might have.
-
Can you elaborate on how this answers the question? Your answer as written is not very clear to me.– John Conde ♦Sep 4, 2018 at 17:10
-
Sure. It appears that Google's Robots.txt file (the file that tells Google what to crawl and what not to crawl on its own domain) may allow the crawling of the /maps. When you add Google Maps to your website, you're basically embedding code from their domain, so crawl rates are controlled by them.– RobSep 6, 2018 at 5:31