What actually matters to users is the time at which something appears on their screen that they can use. That generally means that the page has to finish loading, the CSS and JavaScript has to finish loading, the page has to get rendered, and the images above the fold have to load. For users, overall load time is going to have more of an impact than time to first byte.
Search engines pay attention to how users react to a page. If the page doesn't become usable within about 3 seconds, users start turning back. I've found that any methodology that can decrease the time to the "page load" event to under 3 seconds improves SEO. Further improving performance can increase user interaction, but does not lead to better rankings.
Google has said that they will penalize very slow sites outright, but those penalties only apply if the initial page (as fetched by Googlebot without CSS, JS, and images) takes more than 7 seconds.
It sounds to me like the two plugins you evaluated are not substantially different in terms of their performance benefits. Either of them are likely to be able to get the site fast enough for SEO.
I am skeptical that a caching plugin can decrease overall load time without decreasing time to first byte. Caching plugins work by storing HTML in memory for a page so that it doesn't have to be fetched from the database and assembled by the web application after the first time. The work that a caching plugin is saving all happens before the first byte is sent. After the first byte is sent, the network is usually the limiting factor.