What is the best initial or general setup for the robots.txt to allow search engines to go through the site, but maybe restrict a few folders?
Is there a general setup that should always be used?
Google Webmaster tools has a Section called "Crawler access"
This section allows you very easily to create your robots.txt
For example to allow everything except blog a folder called test your robot.txt would look something like