5

Domaintools when accessing Whois info like this: http://whois.domaintools.com/google.com gives error once free account reaches 20 or so lookups. It then does not return a HTTP 200.

But Google it seems does not get this error and it can crawl all of their URLs. In this case, Google sees a version but visitors may see different version.

I'm not sure how Domaintools allow special access for Google. Isn't that called cloaking?

I need to understand this so that I can use this feature for my sites too.

Improve this question
3
  • "crawl all of their URLs" - You mean, Google is able to "use the tool" to lookup other domains and record the results of these lookups in their results, rather than simply crawling the first 20 pages on their site? If you clear your cookies are you able to clear your 20 lookup limit?
    – MrWhite
    Apr 11, 2014 at 9:15
  • 2
    Domaintools.com records the IP address. It refused after some 20 lookups when accessing through PHP Curl code without accepting cookies.
    – AgA
    Apr 11, 2014 at 10:52
  • On a technical note, this is probably implemented with cookies. The site plants a cookie that counts the number of whois lookups for the user. Googlebot doesn't support cookies and doesn't get this limit. It may have nothing to do with user agent sniffing, which is the usual form of cloaking.
    – Stephen Ostermiller ♦
    Apr 23, 2014 at 14:01

3 Answers 3

Reset to default
2

Are you finding the 20 lookup limit with 20 seperate Google searches ?

Because if you are not and you find the site once and try to search 20 times from that page you found, then it is fine for Domaintools to put a limit on. If not, then maybe Domaintools is using something like First Click Free.

Improve this answer
3
  • First click free is Google's official policy for allowing access to sites that require registration or subscription. It allows content from such sites to be crawled and indexed while ensuring that searchers can access it in limited ways.
    – Stephen Ostermiller ♦
    Apr 23, 2014 at 12:02
  • How do Domaintools implement this? If they single out Google using their IP or Google's user agent string then it is exactly what is called cloaking.
    – AgA
    Apr 27, 2014 at 10:14
  • Simply sniffing out Google isn't cloaking. If you can see the page that Google crawled when clicking upon it in search results without needing to register. log in or pay - how is it cloaking.
    – user29671
    Apr 28, 2014 at 7:59
1

Answer A:

I believe it is more of how they have coded the site for authentication of a user over the search engine.

You can do an If/Else statement within the PHP code. My example is as follows, and I will be using GoogleBot as an example for this.

$googleBot = $_SERVER['HTTP_USER_AGENT']; //  Defines it as GoogleBot Instead

if($googleBot) {

   //  Allows indexing of the full site and goes over the limit of 20
}
else {
   //  Everyone else is only allowed to view the site so many times   
}

Answer B

Google does have to deal with the limit also. There may be a time-limit of how long till Google and yourself can view so many pages again. If so, Google will index pages at that limit also, but it will take a good while to achieve this indexing of the site.

Answer C

I'm going to agree with closetnoc. That is not cloaking. Rules applying to "users" and "bots" are slightly different when you want Google to index your site easily, so that goes back into Answer A with how they are most likely doing it.

Also, you are seeing the same content as Google, but you are only allowed to view it so many times unlike Google.

Improve this answer
-1

This is not technically cloaking. It is not uncommon to allow search engines access to areas of a site that requires a login. Though generally this is discouraged, it is not against the rules. Otherwise, how would these pages get indexed in the first place? How would someone find the sites content even if it requires an account? This is normal business.

There is First Click Free which is Google's compromise to the issue. However, there are very many sites that archive articles, research papers, patents, and so forth that do not engage in First Click Free and yet have all of their pages indexed that either exist behind a login or paywall without penalty. This has been a standard practice since the beginning and is technically not against the rules since the content is exactly the same no matter how it is served- it just requires an account.

Improve this answer
8
  • 1
    This is not normal business, if your pages require a login, you need to show google that same login screen. Or look into something like first click free
    – user29671
    Apr 11, 2014 at 6:53
  • With respect- Google does allow you to show content that requires registration to see. This has been an accepted course of business for a very long time and that fact is self-evident.
    – closetnoc
    Apr 11, 2014 at 14:20
  • No sorry, that is cloaking. It is not allowed.
    – user29671
    Apr 11, 2014 at 15:37
  • Okay. If you insist.
    – closetnoc
    Apr 11, 2014 at 15:48
  • You can learn more about cloaking here : youtube.com/watch?v=QHtnfOgp65Q It is Google who insists that showing different content to bots v users is cloaking.
    – user29671
    Apr 11, 2014 at 15:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.