Here is a robots.txt file that will allow Google, Bing, and Yahoo to crawl the site while disallowing all other crawling:
Some crawlers ignore robots.txt entirely and crawl whatever they feel like. Some crawlers impersonate Googlebot or another legitimate crawler. Some crawlers impersonate browser user agents such as Internet Explorer or Firefox.
There is a procedure for verifying that a user agent of Googlebot is actually a Google crawler. It involves doing some DNS queries against the IP address from which the crawler visited.
There is also the concept of a spider trap which is a place on your website that users wouldn't find, but crawlers would. A spider trap can be used to identify crawlers that are masquerading as browser user agents.