I want to protect my site of scraping, cloudflare helps but not 100%, i have lists of users, i make that lists only in goal for google indexed, problem is because some bad bots can use that lists for easy scraping my all users, i thinked about add captcha for users but not for googlebot, only for pagination on lists, this mean users can see only 1. page of list but when try 2. or any other will see captcha, in same time googlebot will not see captcha at all, my question for seo experts is, is that way allowed or google will read that like cloaking and penalized my site?
1 Answer
I had already published Captcha's impact on SEO for my community (students, clients, and everyone) and from my answer there, I've picked an answer for you below:
Hiding the content behind captchas is NOT GOOD for SEO. Here is why.
- GoogleBot will index that page but cannot crawl through the content.
- GoogleBot assumes this page is a thin content i-e: nothing but a captcha on it.
- Since it cannot crawl, so don't expect any rankings.
- If so many pages are getting indexed like this with thin or just captchas, it impacts overall SEO rankings too.
Besides all this above, your website is NOT penalized just for having captcha-driven content, however, you won't be ranked easily due to inaccessibility to content for algorithms to analyze and rank somewhere.
Wait. It doesn't mean you should stop using captcha to prevent scraping.
Workaround: Serve different version to GoogleBot vs. Users. - This is slightly against Google Policies (falls in cloaking), however, if you do it right, you can still do it. It is confirmed by John Mueller via Google Office Hours SEO Videos (Official Resource).
“From a policy point of view we’re okay with situations where you serve us the full content, and you require a captcha on the user side. If you need to do that slightly differently for Googlebot or maybe other search engines than you would for the average user from our point of view that’s fine.”
