Google Analytics Site Speed tracking utilizes the HTML5 Navigation Timing API. I wrote a fairly detailed answer on it on StackOverflow.
Specifically, it uses these 2 attributes on browsers that have the performance.timing object:
fetchStart attribute
If the new
resource is to be fetched using HTTP
GET or equivalent, fetchStart must
return the time immediately before the
user agent starts checking any
relevant application caches.
Otherwise, it must return the time
when the user agent starts fetching
the resource.
loadEventStart attribute
This
attribute must return the time
immediately before the load event of
the the current document is fired. It
must return zero when the load event
is not fired yet.
That latter event corresponds to the window load event (window.onload). Though this is an important event, this is not the same as when your page becomes usable. So, your page could very well be functioning, but the onload event could be held up by a particularly slow resource on your page.
Does your site include external assets? Advertising? Many social plugins? It's possible that its causing your window load event to be delayed.
It's important to keep in mind that the sampling rate for this feature is by default very low. It's configured to try to sample 10% of visits, but in practice, depending on your level of support, it can be anything between 2%-4% of traffic. So, single outliers can have very significant effects on the quality of the data.
You can get a better idea of how reliable the data distribution is by drilling down into the particular pageview's sampling data to get the distribution
If you're interested in a different data point for your performance, you could use events to construct a custom performance tracker. (It would, however, ruin your bounce rate, as an event would count as an interaction.) For example, maybe the DOM Ready event is a better barometer of when your site becomes usable
if(window.performance && performance.timing)
{
var ms = performance.timing.domContentLoadedEventStart - performance.timing.fetchStart;
_gaq.push(["_trackEvent", "Performance", "Time to DOM Ready","",ms]);
}
(Anecodetally, yes, its very common. Check out some of these times from one of my accounts):
