I have a ruby on rails website that will be in invite-only mode for the next couple of months. Currently I have it set up so visits to any page performs an authentication:

before_filter :authenticate, :except => [:beta] //authenticate checks for a logged in user

But the webpage has a lot of content that I would like to see indexed by search engines, and I was wondering if there's an easy way to allow crawlers to do their work? I am not very knowledgable on SEO related stuff at all, so sorry if this is an suboptimal way to phrase the question.

  • 1
    I think many search engines dislike that. Since a search result the user can't access is worthless.
    – CodeInChaos
    Commented Jun 23, 2011 at 1:59
  • Moreover anything you use to distinguish bots from non-beta users can be manipulated and abused.
    – Joe
    Commented Jun 23, 2011 at 2:14

2 Answers 2


This is a bad idea. This practice, of showing one thing to search engines and another to users, is called cloaking: http://www.google.com/support/webmasters/bin/answer.py?answer=66355

It will cause your site more harm than good, with respect to search engine results; it "may cause your site to be perceived as deceptive and removed from the Google index", according to the above link.

Other suggestions:

  • Can you place marketing/not quite real data outside of your authentication wall that both users and search engines will find useful?
  • Can you place, again, outside of the wall, a subset of the "content that [you] would like to see indexed by search engines", perhaps as a demo?
  • Can you accept not being crawled until you open your site to the general public?

You can do what sites like experts-exchange.com does and use a feature called "first page free". Google will index content that is normally blocked by a pay wall in exchange the publication is required to show the full content of that page to any visitor from Google.

Obviously this means savy users can find your content for free if they keep using Google to enter your site. But it's the only way to get Google to index your content without cloaking and risking a ban from their search results.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.