Yes - all else being equal - decreasing your load time will decrease your average session duration.
Session duration is the time between the first and last interaction hit on your site (an interaction hit being either a pageview or an event that isn't marked as non-interaction).
So GA doesn't care how someone spends that time - whether it's waiting for the page to load or actually browsing.
You should be able to see this using the Page Timings report (Behaviour > Site Speed > Page Timing) and clicking on Technical at the top (under the Explorer tab).
A few caveats:
- this assumes that you've put the tracking tag in the right place, which is immediately after the opening
- exit pages have no time on page, as there is only one timestamp available to calculate with (and therefore bounced visits have no session duration)