Redirects are also only needed when they are hit. Just because your server knows to respond to a specific URL with a redirect, it doesn't mean that it will get any such requests. When you implement redirects you should update your site to not have the links or to point the links directly to the new location. You will get some requests for old URLs from external links and from search engine spiders checking on old URLs, but the majority of users will never use the redirect functionality.
If you find that some redirects are expensive and often used, that performance problem is easy to solve with caching.
There is no reason to be afraid of implementing lots of redirects for performance reasons.