Our site has subscription-only pages - non-subscribed visitors see a snippet preview. As per Google's FCF requirements, your first 5 hits to a subscriber-only pages with .google. as the referrer, you see the full page.

In addition to this, should we whitelist search engine bots so that they can index the full content? I assume this is not required for Google, which can use FCF to index our content, but what about other search engines? Is this considered cloaking?

My gut says that whitelisting bots past the paywall is bad practice., but I wanted to confirm - any evidence or references would be amazing.

1 Answer 1


You should only whitelist bots if you are prepared to offer visitors that come from the search engine the ability to see the content without paying. This is the first-click-free deal: You allow visitors to read some for free, Google gets to index your content, you get traffic from Google.

Search engines will get mad if you let content get crawled and indexed that is behind a paywall. If the visitors can't read it for free, they don't to crawl it and have it indexed. That is a form of cloaking.

  • Sounds like a sort of improvised FCF for search engines that don't support it, which make sense. So, say, whitelist Yandex and also allow visitors referred from Yandex to see full content. Okay. One follow-up though is: having implemented FCF for Google (and I think Bing is working on something similar) should one also totally whitelist their bots?
    – tobek
    Commented Nov 7, 2013 at 15:38
  • See Does Bing support anything like Google's First Click Free program? where I suggest improvising. Commented Nov 7, 2013 at 15:40
  • Yep. To answer my own question in my comment, I found this separate FCF page at Google which does say that you need to whitelist Googlebot past your paywall, in addition to FCF for visits referred to from Google. Good to know.
    – tobek
    Commented Nov 8, 2013 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.