Page speed is a bit of a misnomer. People took the early days of Google using page speed as a factor and seemed to forget what happened immediately after its implementation. I will explain.
When Google announced that page speed would be a factor, a reordering of the SERP results caused a lot of heart-burn. What happened was that Google was ranking vary fast sites far more than sites that were still fast. The SERPs became far too heavy with very fast sites even when these would boost less desirable sites within results. Almost immediately the complaints were loud and strong.
Google re-examined it's metrics and did an mea culpa within weeks. As it turned out, acceptable speed sites were being shutout from the top 10 results. "Foul!" and "Not fair!" was the cry from many site owners and they were right. If your site is within a normal response range or greater, there is no boost. However, if your site is below a normal rate, then there is a down-grade. Your site is measured overall, however slower pages can have an effect short of the obvious reason- content size. There is a measure that makes larger pages loading slower more acceptable that smaller pages loading slower.
It is all a matter of whether your site is within an acceptable range as determined by the measures of all the sites that Google has indexed.
Making your site as fast as possible and as lean as possible is a good thing and you should do just that. But Mother Superior Google is not slapping your wrist with a ruler if you get a B or even a C in class. If you get an F however, WATCH OUT!