If you are directing your users to 404 pages on other sites, your users are not going to be happy. For the sake of your users, you remove the broken links.
I was involved in the open directory project (dmoz.org) several years ago. One of the biggest problems that a directory site faces is broken links. The quality of the directory is based on how good its recommendations are. Broken links are the worst kind of recommendation.
I would recommend using link checking software to automatically scan all the links and find the ones that are broken.
From an SEO perspective, anything that is a usability problem is also an SEO problem these days. When users start preferring other sites over yours from search, your rankings fall.