Can I rely on Google Lighthouse reports (especially for mobile sites)?

Question

We have started to use Lighthouse to track the improvements we make to our sites. While this seems to work quite well for desktop sites, i.e. we see the values improve over time and as we make changes, for mobile sites the values remain consistently low. We do repeat the tests and use the best of three, but still.

Below, we have the results of the New York Times mobile site that according to Lighthouse appears to perform badly vis-a-vis the desktop site. The other two are sites of ours, the main site and the third one being our own.

Browsing the site (as well as the NYT, of course) this apparent bad performance cannot be felt at all.

The test procedure:

• run same test three times for each site
• mobile
• no PWA
• incognito mode

Now, while initially enthusiastic about Lighthouse's capability to evaluate a site by attributing aggregated figures that are easy to digest by management-type people, we have the impression that they are not actually that useful as they don't correspond to the users' reality and don't change even though we make changes.

Also, this being a Single Page Application, the first load of the page may take some more time, but any further navigation is quasi-instantaneous. We could not find a Lighthouse feature to take this into account.

Note this is a repost of my question on Stack Overflow.

Answer 1

Rely on it for what, is the key question. If you want a simple reporting metric for a non-technical audience, this isn't it. The scoring system is fairly complex, as are the things it's trying to measure.

In general, though, like any tool you have to invest some time to fully understand what it measures and how. There are a lot of reasons why Lighthouse results are variable; see this page for some in-depth discussion.

Something like CrUX, which uses actual user data, might be better suited as a business reporting tool. It's measuring many of the same things, and while the language is technical (first contentful paint, etc.) the underlying concepts are very tangible and easy for a non-technical audience to grasp. There's a basic dashboard set up here.