2

My website pages are getting 8 to 10x traffic from bots than from genuine users. I'm seeing nearly 90 different bots in my logs. They are all "good bots" by definition. Mostly search engine crawlers, seo indexers like ahrefs and digital ad networks. My problem is the cost. I use Google Maps and Places APIs and am charged a unit cost per API call. I know how to block bots from crawling pages and that is 100% successful for me as is my ability to block them from making chargeable API calls.

The question I have is what is the SEO impact of denying a bot from seeing content provided by the APIs? I am presenting text restaurant location data obtained from Google Places and Yelp Fusion and showing locations on an embedded Google Map.

I'm considering implementing a strategy of allowing only select search engine bots to hit the APIs. Do I need to do this or can I just serve up a generic map image and dummy restaurant location data to bots without SEO penalty? I show a page for each restaurant with the address, city, state, postal code and phone number obtained from Google Places or Yelp Fusion API. Yelp is free but I'll exceed their quota if I fulfill all bot requests.

| improve this question | |
  • Are the JavaScript APIs or are they server side? If they are JS based, most bots won't execute them. Out of all the bots, only Googlebot implements enough JS to use your API if it is JS based. – Stephen Ostermiller♦ Apr 28 '19 at 11:21
  • All my API calls are server side. Interestingly, the bot traffic is 100% excluded by Google Analytics and a high percentage of it is not counted by Google Maps API. I have not dug into which bots count or not. – Art Apr 29 '19 at 19:00
  • Google Analytics tracking snipped is JavaScript based, so it isn't surprising that most of the bots get excluded from GA. – Stephen Ostermiller♦ Apr 29 '19 at 19:44
1

yes, it would be a right strategy to allow only top search engine bots for crawling & indexing your web pages.

I need few more inputs for your query about 'generic map image and dummy restaurant location data'

Did you mean of having dynamic data for each locations?

I would suggest to have a static pages (with unique page URLs) for each location with relevant restaurant data.

Which can be segregated by: locations, categories & sub categories, ratings etc.

Additionally, you can trigger API calls based on user action

Like a CTA button for making the API calls, if possible

| improve this answer | |
  • Roshan: I have a dynamically generated page for each restaurant location, there are about 150,000 in my db. There's a Google map displayed along with the restaurant's address and phone number. The app doesn't require any other data. The restaurant info is pulled using Google Places API. Their terms of service specifically and emphatically don't allow saving any data, static pages wouldn't be an option. I was suggesting serving generic and dummy data to the bots instead of making an api call so at least the page layout would be the same. Your CTA idea works for another challenge I have. Thanks. – Art Apr 29 '19 at 19:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.