If you really want to control the indexing of your site either to block it or promote it, the process is the same. Yes, you have the Robots.txt to help you out for the majority of search engines (that support it, a.k.a. the legit ones). But for the really major engines, like Google, Yahoo, and MSN (not sure about Ask), you get tools from them that allow you to control what is indexed to a much finer digree than Robots.txt ever will. For Google it was the Webmaster Tools, for Yahoo and MSN I forget what they were called, but you can find their tools as well. Sadly though, the Yahoo and MSN tools are way behind Google's, but should still do a good job for basic needs.
Using the combination of Robots.txt and the tools I've successfully kept a group of related websites from ever being indexed (they were being kept private), but I could have also promoted their ranking as well if I needed to through the same tools.