5

My website has pages for around 1000 companies. Without an account, I only allow users to view 5 company pages a month (I track them by cookie). And Googlebot can crawl my site freely since it does not use cookie.

To avoid Googlebot thinking I am doing cloaking, I am specify the content of my company pages as paywall content (following this guide). The problem is now when I search the company pages of my website on Google, the search result only shows the page title and my website's default description. It does not show content of each company page.

I think the problem is I list the content of my company pages as paywall. My question is: What is the best way to let Googlebot crawl and display the content of my website in the search results without making Googlebot think that I am cloaking? (when the users click on the search result link and visit my website, they are still subject to the monthly quota if they don't have an account).

2
  • Please clarify what this means: It does not show content of each company page. Aug 20, 2021 at 17:21
  • Whitelist Googlebot's known IP address ranges on your web server.
    – Kate
    Aug 20, 2021 at 18:46

1 Answer 1

5

Google says in their flexible sampling documentation:

Enclose paywalled content with structured data in order to help Google differentiate paywalled content from the practice of cloaking, where the content served to Googlebot is different from the content served to users.

If you are using structured data to tell how the content is limited for users, then you may give Googlebot full access based on either the User-Agent header or its IP addresses. You won't be penalized for cloaking when implementing flexible sampling despite basing decisions about access partly on whether or not it is Googlebot.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.