First, I'll make a grand apology to all the professional webmasters out there, but for this OP, I have one golden suggestion:
Violate search engine guidelines
And I mean do it to the point where the important content is in complex javascript and the content robots can crawl is not in proper HTML. This includes a bad description tag, a bad title tag, etc. Heck, maybe turn the entire content into just a video made in flash, or show the entire content as just one image. That would really make the search engine crawler cringe.
I'll show by example in code:
Here's a way to get something indexed:
<!DOCTYPE html>
<html>
<head>
<title>Web page</title>
<meta name="description" content="This is a wonderful web page">
</head>
<body>
<h1>A wonderful web page</h1>
<h2>By John Smith</h2>
<p>This is a wonderful page. ya de ya de ya de ya de ya de ya de</p>
<p>This is wonderful. ya de ya de ya de ya de ya de ya de</p>
</body>
</html>
Ok, I admit, the text isn't perfect, but you understand what I mean.
Now if you want to hide it from crawlers and do it the simple way, you can try this:
<!DOCTYPE html>
<html>
<head>
<title>Private</title>
</head>
<body>
<img src="mywebsite.jpg" width=1024 height=768>
</body>
</html>
then make an image named mywebsite.jpg and include all the text in that, not in the html shown above. Then you need to protect mywebsite.jpg by making a watermarked version of it to the users who are unauthorized to see the real thing. Simply compare user agent strings or ip addresses against those you allow/disallow for the image. This type of thing can be done in .htaccess with some rewrite rules.
For example, to force googlebot to see the watermarked image instead of the real thing, use these rules:
RewriteCond %{HTTP_USER_AGENT} ^googlebot$ [NC]
RewriteRule ^mywebsite.jpg$ specialrobotimage.jpg [L]
I'm assuming here that mywebsite.jpg is your real website as an image and specialrobotimage.jpg is the watermark or the image as a message stating only real users are allowed to see the information. Also, the rules assume everything is in the same folder.
robots.txtfile with the right contents can keep all the legitimate bots out. The hard part is the rest. A lot of them can be kept away if the address of the site is hard to find.