Google currently uses 2 bots for crawling for data, these are:
Both bots have the ability to read screen size and will take into account resources that are not being loaded for ranking purposes, for example say you
display:none on a mobile device using a media query but on the desktop you do a
display:block then Google will or least should account this resource only for the Desktop. Now in terms of page-speed I would assume or least hope that Google takes into resources that are hidden via display:none into screen measurements though this may open up a can of worms of people exploiting the page speed algorithm and I believe this is why I'm unable to find concrete evidence to support this, most of Google's bot algorithm's are kept extremely secret, and no one truly knows for certain on things such as this.
Now assuming that Google doesn't take into account things like
display:none when browsing a site for site speed then it should be still possible to have a fast site that all resources are loaded, it would be worth your while to 'Test' all resources loading at once, there are sites many large sites that load hundred of resources per a page and these are fast down to the hardware they use, setup they use, and the optimisation. Even with doubling up on some resources such as sprites made for mobiles and one made for desktop then you should be able to get the site working fast with both loading. Personally I'd setup a test and have the site load all resources and attempt to get it down to 1.5secs.
SOURCE: Building Smartphone-Optimized Websites
Responsive web design is a setup where the server always sends the
same HTML code to all devices and CSS is used to alter the rendering
of the page on the device using media queries. Our algorithms should
automatically detect this setup if all Googlebot user agents (both
Googlebot and Googlebot-Mobile) are allowed to crawl the page assets
The max-width value of 640px shown above is an example, not a
requirement. Our algorithms look for max-width values that can be
reasonably expected to refer to smartphone screen resolutions, and we
will try to monitor what typical mobile websites use and may update
our algorithms accordingly in the future.