Without knowing more about the website in question, I'd say time on page isn't necessarily the best indicator. People may leave a page quickly because they can skim and see it doesn't meet their expectations, but they may also leave a page quickly because they found exactly what they needed and no longer need to stay on that page (or even your website). And, different people have different opinions on what "optimizing" a page may mean - it's not clear whether you mean perform SEO to attract visitors, or improve UX on a page, or something different.
So, you really need a way to measure whether or not visitors are completing actions that matter. Rather than guessing at which content you should update, it would be worth your time to sit down and map out your website's goals. These will usually be a mixture of your goals (get people to subscribe to your email list, read your content, buy a product) and the visitors' goals (find the answer to a question, buy a product that meets their needs). Once you have those mapped out, you can determine whether to record each one as a Goal or an Event.
Typically Goals are the big actions - such as buying a product or subscribing to an email list - that are easy to quantify. Typically Events are smaller actions that may lead to a purchase or higher engagement - such as scrolling halfway down the page.
Start measuring goals and/or events in Google Analytics. Once those are set up and have run for at least a month or two, you can then compare your pages. The pages that have the highest numbers of visitors, but the lowest numbers of conversions (either events or goals), are the ones you should target - people are actively visiting them, so making the pages easier to read or with clearer calls to actions should lead to the most improvements in conversion.
If instead you are only talking about SEO optimization, it would still make sense to set up the goals and events, but in the meantime you can start with the pages that have the highest numbers of visitors. These again are the ones search engines are already picking up, visitors are coming to, so improving the pages for both search engines and human visitors should give you the most bang for your buck.