3

I am trying to protect my back-end resources using re-captcha from misuse from bots however am not sure of its implication on SEO. There is little information clarifying about this issue. From re-captcha documentations they just say re-captcha protects one from bots

Here is how I theorize that re-captcha works:

Before making an API call from my expensive API operation...it performs an action to re-captcha and gets the score. If the score is less than 0.5 the user is redirected to /blocked.html if the score is above 0.5 the API operation pass and gives results.

Will this block google-bot as it tries to index my page?

With the introduction of firebase app check which somewhat does this got me thinking if its possible ? https://firebase.google.com/docs/app-check/web/recaptcha-provider

1 Answer 1

3

Adding a captcha check before directing the user to the next section of your site will more than likely prevent the Googlebot from continuing. I say more than likely as Google provides reCaptcha and MAY have implemented a bypass to allow the Googlebot to access resources protected by reCaptcha, however this is unlikely. The best way to ensure access is to check to see if the connection is being made by Googlebot and only show reCaptcha if it isn't Googlebot making the connection. The vast majority of legitimate search engine crawlers will identify themselves through the user agent string and will also follow the robots.txt file rules for what paths they can and shouldn't crawl. If you want to allow approved bots access for crawling in order to index the subsequent pages then you can do a check on the user agent string to check if it is Googlebot. You do need to bear in mind that the user agent string is by no means a reliable indicator of the bot as you can program any user agent string into your bot, as such Google publishes a list of IP address ranges which legitimate Googlebot connections will come from, and while this range of IP addresses can and often does change it is published as a JSON file so that you can download the file at semi-regular intervals and update your IP checking validation rules as well. The file can be found at https://developers.google.com/search/apis/ipranges/googlebot.json.

The only thing I would add from here is to be absolutely sure the effort involved in controlling block access to a given route in your site is worth the effort. Once you start down that track it is a never ending battle to block bots as you will never be able to block them all, and a lot of the bad bots out there will actually use random IP addresses and appear as though they are a legitimate connection from a regular user, rather the often preferred choice rather than blocking bots is to optimize performance throughout your processing and delivery pipeline to ensure that a bad bot doesn't cause undue issues by accessing an expensive API call.

1
  • That was detailed i appreciate. Wasn't aware of the ip ranges will implement those on my servers asap. Managing real and fake google bots has been a nightmare.
    – xaander1
    Commented Dec 30, 2021 at 1:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.