There are so many factors at play it's really tricky to suggest without knowing a lot more about your site and the way it's used.
If you're looking at a large site I would suggest breaking down the data by location, it's likely that one geographic region is skewing the results. Also look for pages which contain very large images, or large numbers of images but are infrequently accessed.
GA is prone to results being skewed by a small number of outlying data points, especially in custom reporting.
Caching will only get you so far, you should consider a CDN, read Yahoo!'s Best Practices for Speeding Up Your Web Site and look at page level optimisation in addition to site wide optimisation. i.e. if you're loading lots of scripts, some may only be needed on certain pages. CSS sprites can also offer tremendous benefits to medium and high traffic sites.
See Google support's Interpret Site Speed (my emphasis):-
In speed analysis, looking at the average doesn't always tell the
whole story, because a few outliers can skew the average. That’s why
we recommend you look at the Performance tab to understand the
distribution of your load/execution times.
And About Site Speed (my emphasis):-
On the Page Timings report, the Explorer and Map Overlay tabs contain
two sections each: the Site Usage section and the Technical section.
Site Usage shows your speed data at a high level in context of other
Analytics metrics. The Technical section shows your data in more
detail, providing a breakdown of network and server time metrics. The
six sections on the Performance tab display different metrics grouped
by speed ranges.