4

My website consists of multiple pages, each page is for a company. I plan to allow users that are not signed in to view the company pages, but limit to 10 companies a month. I will use a cookie to track this. (I know users can delete the cookie but it is not the point of this question.)

However, I would like to allow Google crawl my website without limits for SEO purposes. Is it doable? I know Google cannot crawl pages which require users to sign in. Not sure whether Google can crawl pages which do not require users to sign in but which have a limit on page views.

10

Googlebot will be able to crawl your site fully because it never sends any cookies. There is, however, something else you need to do.

Google expects sites to implement limits on the amount of content that users can view per month. Google has a policy about allowing sites in the search results even though they are limited. They call this policy flexible sampling. Your plan of allowing 10 pageviews per month fits nicely into what Google expects. Google says 6-10 free pages per month is expected.

Other than making sure your site is crawlable, the other requirement is informing Google that you are limiting users' free page views. This is done with structured data for subscription and paywalled content. You need to add this structured data to your pages.

4
  • 1
    thank you for your answer. I saw another answer (stackoverflow.com/a/61955273/15296954) for a related question mentioning that we can collect client's data like timezone, fonts installed, screen resolution etc to uniquely identify unsign-in users and impose monthly limit based on that data instead of cookie. I wonder whether imposing the monthly limit based on client data affects Googlebot's ability to fully crawl my website?
    – VII
    Aug 6 at 14:20
  • 2
    That technique is called "fingerprinting." Googlebot is likely have the same fingerprint every time it downloads a page. If you are limit the number of pageviews per fingerprint, it would stop Googlebot from crawling. You would have to do something special in that case. You could whitelist the Googlebot user agent, far example. When you make decisions based on crawler user agent it is called cloaking. Normally this could get your site banned, but in this case with the paywall structured data it would be fine. Aug 6 at 14:26
  • 2
    See the bottom of the flexible sampling document for confirmation about differentiating cloaking from allowed flexible sampling. Aug 6 at 14:33
  • Hi Stephen, I hope you don't mind me asking a follow up question here. I have seen quite a few articles saying that Google is against fingerprint tracking (example: digiday.com/media/…). I wonder whether Google punishes search ranking for websites which use fingerprint tracking?
    – VII
    6 hours ago
4

I will use cookie to track this...

If you are using cookies to track whether unsigned-in users have reached their page-view limit then there is nothing extra you need to do to allow Googlebot unrestricted access, since Googlebot does not send cookies.

1
  • 2
    You are also making it easy for knowledgeable users to bypass your paywall.
    – user253751
    Aug 6 at 9:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.