Our site has been doing fairly well in terms of load speed and so I don't want this to bring that down as a slower page load speed impacts SEO visits.
Almost certainly no. Every network request is sent using packets usually of 1500 bytes. If the request is under 1500 bytes (including the URL, cookies etc) then it makes no difference whether it's 30 bytes or 1300 bytes. Even if a request spans 2 packets the difference would be negligible - that's definitely a micro-optimisation you don't need to do unless you're Google.
So keep URLs reasonably short (less than 100 characters) for the sake of usability, but there's absolutely no difference in speed.