It is possible that the time on site is now dropping. But did you also check the bounce rate? But better page performance is never a decline.

Tip: The absolute value of "time on site" is useless. You have to check your values against your competitors in your specific niche.