It will not, or at least it makes sense to block them. Just compare:
1) blocking at the firewall level is close to the the kernel, therefore very fast
2) allowing the attacker to pass to the site involves apache, perhaps php and 404 error generation. Too much CPU and network resources.
If you have have millions IPs to block there, then it will slow down the server, but on the other hand, to generate several errors (e.g. wp-admin, wp-login, xmlrps etc) per each IP for the same millions of IPs... It worth to block.
Also, fail2ban is quite helpful: I believe it can remove from the black list IPs which did not tried to attack for certain period of time.