Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Our site has subscription-only pages - non-subscribed visitors see a snippet preview. As per Google's FCF requirements, your first 5 hits to a subscriber-only pages with .google. as the referrer, you see the full page.

In addition to this, should we whitelist search engine bots so that they can index the full content? I assume this is not required for Google, which can use FCF to index our content, but what about other search engines? Is this considered cloaking?

My gut says that whitelisting bots past the paywall is bad practice., but I wanted to confirm - any evidence or references would be amazing.

share|improve this question
add comment

1 Answer 1

up vote 0 down vote accepted

You should only whitelist bots if you are prepared to offer visitors that come from the search engine the ability to see the content without paying. This is the first-click-free deal: You allow visitors to read some for free, Google gets to index your content, you get traffic from Google.

Search engines will get mad if you let content get crawled and indexed that is behind a paywall. If the visitors can't read it for free, they don't to crawl it and have it indexed. That is a form of cloaking.

share|improve this answer
    
Sounds like a sort of improvised FCF for search engines that don't support it, which make sense. So, say, whitelist Yandex and also allow visitors referred from Yandex to see full content. Okay. One follow-up though is: having implemented FCF for Google (and I think Bing is working on something similar) should one also totally whitelist their bots? –  tobek Nov 7 '13 at 15:38
    
See Does Bing support anything like Google's First Click Free program? where I suggest improvising. –  Stephen Ostermiller Nov 7 '13 at 15:40
    
Yep. To answer my own question in my comment, I found this separate FCF page at Google which does say that you need to whitelist Googlebot past your paywall, in addition to FCF for visits referred to from Google. Good to know. –  tobek Nov 8 '13 at 16:38
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.