Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a ruby on rails website that will be in invite-only mode for the next couple of months. Currently I have it set up so visits to any page performs an authentication:

before_filter :authenticate, :except => [:beta] //authenticate checks for a logged in user

But the webpage has a lot of content that I would like to see indexed by search engines, and I was wondering if there's an easy way to allow crawlers to do their work? I am not very knowledgable on SEO related stuff at all, so sorry if this is an suboptimal way to phrase the question.

share|improve this question

migrated from stackoverflow.com Jun 23 '11 at 10:30

This question came from our site for professional and enthusiast programmers.

1  
I think many search engines dislike that. Since a search result the user can't access is worthless. –  CodeInChaos Jun 23 '11 at 1:59
    
Moreover anything you use to distinguish bots from non-beta users can be manipulated and abused. –  Joe Jun 23 '11 at 2:14

2 Answers 2

This is a bad idea. This practice, of showing one thing to search engines and another to users, is called cloaking: http://www.google.com/support/webmasters/bin/answer.py?answer=66355

It will cause your site more harm than good, with respect to search engine results; it "may cause your site to be perceived as deceptive and removed from the Google index", according to the above link.

Other suggestions:

  • Can you place marketing/not quite real data outside of your authentication wall that both users and search engines will find useful?
  • Can you place, again, outside of the wall, a subset of the "content that [you] would like to see indexed by search engines", perhaps as a demo?
  • Can you accept not being crawled until you open your site to the general public?
share|improve this answer

You can do what sites like experts-exchange.com does and use a feature called "first page free". Google will index content that is normally blocked by a pay wall in exchange the publication is required to show the full content of that page to any visitor from Google.

Obviously this means savy users can find your content for free if they keep using Google to enter your site. But it's the only way to get Google to index your content without cloaking and risking a ban from their search results.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.