There are a number of things that could cause this, and they are not necessarily a problem:
- Streaming media (videos, audio, etc.) on your page. This can add to the overall load time, as until it's finished streaming, the loading time keeps counting.
- Lazy loaded images. Like streaming media, if you're lazy-loading images as the user scrolls, this can keep adding time to the page load - if you load the page, then wait a while before scrolling you'll see that the wait time is included in your page load time.
- Other delay loaded resources. If you've got some script that fires after an interval and loads further resources (like a survey or interrupt) then the delay will be included in the page loads.
If you use the "Network" tab of Chrome or Firefox's Developer Tools, set to "All" you should be able to see something like this:

You can see the bulk of this site loaded in 560ms, but the total load time was 2.9 minutes, and there are a couple of outliers between 10 and 20 seconds, and then a much longer delay before the final images - you should be able to identify what's loading either slowly (with a long bar) or just being called very late.
If you find that a particular resource is taking a long time to load, hovering over it will reveal more information about that particular request:

Which might help you identify ways to improve the performance. For example if there's a large delay between a request being queued and it actually starting, you are making too many simultaneous HTTP/1 requests to a single server, and should consider bundling if possible, while if your waiting (TTFB) or content download seem high for the size of the request, you might want to consider caching strategies.