Robots.txt is a file that other websites, ISP's, and search engines use to "ask you" whats ok to visit. It allows you to whitelist or blacklist all or specific bots from areas of your realm. It's like a treaty. It's a promise. Good things keep the promise, bad things do not.
As far as search: I agree that in the past it was not good practice to allow robots to hit search. Nowadays, allowing Google to hit up search may work out well; at least in certain niches; and you don't even need search caching.
The robots.txt's across our platforms vary, but we always leave the search disallow commented out (AKA robots allowed to search, but it's ready to be uncommented if needed). There are a few reasons:
- Fills in SEO - sometimes you will see search results popup for category niches you missed.
- Fills in LSI - helps you create organics from organics, automagically
- May Help RDF - this is edge but allowing G to search may expose rich snippets faster
- Makes Authority - See a search page SERP result dominating organics? Turn it into a lander to gain PR
- Helps G Understand - between tab-search in address bar, analytics search teach, and webmaster tools query string parameters, G will understand and help.
Look for areas in G analytics, G webmaster tools, and other G areas to set up search now and in the future.