Although, not a primary ranking factor, we would recommend that you clean them up so there are no 404s or missing pages.
If an auditing software can detect these errors, then you can bet that Google can, too.
The usual way to "block" access to these files would be to serve a 403 Forbidden.
For example, using mod_authz_core on Apache 2.4:
Require all denied
If you specifically want to respond with a 404 instead then you can use mod_rewrite. This needs to go near the top of your config file. For example: