1

We are using a cookie on our website which is to fetch some information through an API. The cookie becomes invalid after a very small duration.

Google Bot uses the same cookie again and again to crawl our website and we see numerous of exceptions and errors in the API server.

Please suggest me the way to handle the scenario gracefully.

  • Googlebot doesn't crawl using cookies at all as far as I know. I've also never seen Googlebot try to use an API. I'm surprised to hear this. Can you tell us more about this situation? How did you publish the API in a way that Googlebot finds it? Are there links to get data via the API? Is this part of some AJAX rendering for your page? – Stephen Ostermiller Nov 19 '18 at 11:21
  • @StephenOstermiller Yes, we are rendering one section of the page as react component through AJAX call followed by an API. There are links to get via API. Can you please redirect me to a document which explains about Google-Bot not using cookie or API to crawl a page. – abhishek shrivastava Nov 19 '18 at 14:18
2

As far as i know Google is not using cookies or API when crawling a website, the issue should be something else check again your API errors or servers logs to determine the issue.

Google states:

HTTP cookies, local storage, session storage are cleared between page loads.
All features requiring user permissions (like Notifications API, clipboard, push, device-info) are disabled.

Read here more about Googlebot rendering

And a moz guide here

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.