It is possible that the time on site is now dropping. But did you also check the bounce rate? But better page performance is never a decline.
TippTip: The absolute value of Time"time on sitesite" is useless. You have to check your values against your competitors in your specific niche.