3

What is the potential list of origins of the host domains through which google bot crawls?

e.g.: Like Google cache uses the following domain to crawl the APIs: http://webcache.googleusercontent.com/

I want to add those google domains into my APIs, so Google bot easily crawls my website. So bot or crawler don't get cross origin error.

  • Google cache is now a crawler. Google cache displays your cached pages on webcache.googleusercontent.com but no crawler ever runs from those domains. Users that view the cached page may request files from your live website with that domain as a referrer. Are you worried about that cross domain errors for that situation? – Stephen Ostermiller Sep 6 at 22:14
  • I don't think you need to worry about Googlebot encountering any cross-domain errors that that users wouldn't also see. – Stephen Ostermiller Sep 6 at 22:15
2

According to the official documentation, Googlebot always crawls using either the googlebot.com or google.com domains.

It's worth mentioning that the documentation also recommends using a reverse DNS lookup on the bot's IP to fully verify the bot's authenticity.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.