4

I'm working on my site and as it require users to log in it will be hard to Google to index the sites, because to see 90% of the content you have to be online.

So I've made a script which looks for the ip that enters the site if it is between 66.249.66.1 to 66.249.71.206. So if the ip is between this range, i set "the google bot" as online and it will be able to see all pages a normal user would see.

Is this a good idea? Are there more ip ranges? Can I trust these ip ranges?

| improve this question | |
  • The IP range is currently 66.249.64.0 to 66.249.79.255, but as has been mentioned, you can't trust ranges. – Oskar Skog Jul 21 '17 at 22:28
6

This is not a good idea and no, you cannot trust these IP ranges. The IP addresses used by Google are not public. But some/most search engine crawlers can be identified by doing a reverse DNS lookup on the IP address.
A googlebot example: 66.249.64.0 has a PTR record to crawl-66-249-64-0.googlebot.com, and any IP with a PTR record to a subdomain on googlebot.com is an IP address used by googlebot.

What you are doing is showing one set of content to Google and another to the actual user. This is heavily frowned upon and is called cloaking.

You should watch Matt Cutts' definitive cloaking video.

The best option is to take a subset of that content that you are willing to be publicly visible and make a portion of the site that exposes this content to the search engines and users, and if users want to see more they'll have to login.

| improve this answer | |
  • So you mean that I can't index the content that a user will see when they login? – Kilise Feb 13 '13 at 13:19
  • 1
    Not if you must login to see it. The best option is to take a subset of that content that you are willing to be publicly visible and make a portion of the site that exposes this content to the search engines. – joesk Feb 13 '13 at 13:21
  • 1
    Alright thanks a lot! I have to try to create a script that shows the user (including google) like 20% of the content , and if he want to see more he has to login. Sounds good? – Kilise Feb 13 '13 at 13:24
  • 1
    Yep, that's pretty much it. It help to make sure you have the relevant keywords/phrases etc in the visible content for the search engines. – joesk Feb 13 '13 at 13:55
  • 1
    @joesk is 100% right, many sites will show part of the content to get around this. The only company that is able to get away with cloaking their content currently is Google itself (some of their result content requires a G+ log in). – Ben Hoffman Feb 13 '13 at 16:26
3

If you want to give Google access to restricted content, you can use First Click Free by Google.

First Click Free is designed to protect your content while allowing you to include it Google's search index. To implement First Click Free, you must allow all users who find your page through Google search to see the full text of the document that the user found in Google's search results and that Google's crawler found on the web without requiring them to register or subscribe to see that content. The user's first click to your content is free and does not require logging in. You may, however, block the user with a login or payment or registration request when he tries to click away from that page to another section of your content site.

| improve this answer | |
3

What you do is not a good idea and can be penalized as cloaking.

Till 1st October 2017 the best practice was the First Click Free as mentioned in a previous answer. However since October 2017 this has changed.

Now google uses Flexible Sampling for paywalled or otherwise not freely available content.

Basically Google lets the publishers decide how much content they will offer without restrictions, but they should mark up their content accordingly. So Google understands which content is protected and doesn't penalize the site for cloaking. A publisher can decide to offer a limited number of pages or just portions of pages for free and have the rest restricted.

Google indexes all restricted pages if the robots can see them. However the fact that they are protected may affect their ranking in ways only Google knows.

| improve this answer | |
  • 1
    This question needed an updated answer given the recent changes from Google. – Stephen Ostermiller♦ Nov 6 '17 at 20:23
  • Thanks for keeping this subject updated with latest support. – Kilise Nov 7 '17 at 7:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.