I would like to somehow be able to block all search engines except Google, Yahoo & Bing (and their related sites like Google Images) from crawling my site as they consume a lot of server and bandwidth but don't bring any traffic.
Is this easily done or difficult? It would be good if someone maintained a list of small search engines that could be pasted into a robots.txt file to block them.
Also, I realize I cannot block crawlers that ignore the robots.txt or sites from surreptitiously scraping and crawling, but that is not what I want. I just want to block all the Altavistas, Hotbots, Lycos (do these even still exist) and the university experiment crawlers from wasting my time.