Is limiting the page views from users but allowing Googlebot unlimited access considered cloaking?

Question

Domaintools when accessing Whois info like this: http://whois.domaintools.com/google.com gives error once free account reaches 20 or so lookups. It then does not return a HTTP 200.

But Google it seems does not get this error and it can crawl all of their URLs. In this case, Google sees a version but visitors may see different version.

I'm not sure how Domaintools allow special access for Google. Isn't that called cloaking?

I need to understand this so that I can use this feature for my sites too.

Answer 1

Are you finding the 20 lookup limit with 20 seperate Google searches ?

Because if you are not and you find the site once and try to search 20 times from that page you found, then it is fine for Domaintools to put a limit on. If not, then maybe Domaintools is using something like First Click Free.

Answer 2

Answer A:

I believe it is more of how they have coded the site for authentication of a user over the search engine.

You can do an If/Else statement within the PHP code. My example is as follows, and I will be using GoogleBot as an example for this.

$googleBot = $_SERVER['HTTP_USER_AGENT']; //  Defines it as GoogleBot Instead

if($googleBot) {

   //  Allows indexing of the full site and goes over the limit of 20
}
else {
   //  Everyone else is only allowed to view the site so many times   
}

Answer B

Google does have to deal with the limit also. There may be a time-limit of how long till Google and yourself can view so many pages again. If so, Google will index pages at that limit also, but it will take a good while to achieve this indexing of the site.

Answer C

I'm going to agree with closetnoc. That is not cloaking. Rules applying to "users" and "bots" are slightly different when you want Google to index your site easily, so that goes back into Answer A with how they are most likely doing it.

Also, you are seeing the same content as Google, but you are only allowed to view it so many times unlike Google.

Answer 3

This is not technically cloaking. It is not uncommon to allow search engines access to areas of a site that requires a login. Though generally this is discouraged, it is not against the rules. Otherwise, how would these pages get indexed in the first place? How would someone find the sites content even if it requires an account? This is normal business.

There is First Click Free which is Google's compromise to the issue. However, there are very many sites that archive articles, research papers, patents, and so forth that do not engage in First Click Free and yet have all of their pages indexed that either exist behind a login or paywall without penalty. This has been a standard practice since the beginning and is technically not against the rules since the content is exactly the same no matter how it is served- it just requires an account.