3

What is the potential list of origins of the host domains through which google bot crawls?

e.g.: Like Google cache uses the following domain to crawl the APIs: http://webcache.googleusercontent.com/

I want to add those google domains into my APIs, so Google bot easily crawls my website. So bot or crawler don't get cross origin error.

2
  • Google cache is now a crawler. Google cache displays your cached pages on webcache.googleusercontent.com but no crawler ever runs from those domains. Users that view the cached page may request files from your live website with that domain as a referrer. Are you worried about that cross domain errors for that situation? Commented Sep 6, 2020 at 22:14
  • I don't think you need to worry about Googlebot encountering any cross-domain errors that that users wouldn't also see. Commented Sep 6, 2020 at 22:15

1 Answer 1

2

According to the official documentation, Googlebot always crawls using either the googlebot.com or google.com domains.

It's worth mentioning that the documentation also recommends using a reverse DNS lookup on the bot's IP to fully verify the bot's authenticity.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.