0

I have a website with public limited access, via maximum number of requests per IP per day. Though I don't want to block Google crawler. Just checking for the Google user agent default headers is not a solution, because it would be trivial to emulate and hence bypass the site's limitations.

Hence, can I set HTTP headers on Google user agent with Google Search Console, to embed a key therein?

1
  • 3
    As far as I know, key embedding is not supported. As an alternative, if you are worried of people faking Googlebot's User Agent, you could double-check by verifying it comes from Google ASIN/ASN (15169). If you have a WAF like CloudFlare, it's easy to enable such a check. Otherwise, you'd need to check against their IP ranges. Commented Jun 7 at 19:13

1 Answer 1

0

Eventually I developped a NPM Express package to detect if request was coming frm Google, based on known IPs

https://www.npmjs.com/package/express-is-googlecrawler

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.