Google have a separate mobile crawler (Googlebot-Mobile) in addition to the regular Googlebot. Googlebot-Mobile will look at content that is intended both for "feature" or "dumb" phones (e.g., WAP, etc.) and smartphones (see announcement here).
If you're serving different content based on user-agent detection, either by redirecting to a separate site (m.example.com, etc.) or on same URLs, Google likes you to use the
Vary header to help them discern which content is meant for what device.
Similarly, if you're using responsive design, Google can detect CSS media queries and determine whether they're set to address common phone viewport sizes. From their developer guide:
"Our algorithms look for max-width values that can be reasonably expected to refer to smartphone screen resolutions"
They've retired Instant Preview now, but when it was still around you could actually verify that this works by noticing that the preview of a responsive site in the mobile index would show the layout as it would appear on a smartphone.
Does any of this affect ranking? All things being equal, I'd expect that a mobile optimised page would rank more highly in mobile search than a non-mobile optimised equivalent.