Add visual indicator to score to warn when it was run with non-default settings

[av3nger]

I am seeing inconsistent ratings for same metrics when doing scans from PageSpeed Insights vs a Lighthouse scan from Chrome. For example, a scan from PageSpeed Insights is rating the First Meaningful Paint at 1.2 seconds and marking it as a warning (see first screenshot), while a scan from Chrome is marking the same metric even at 1.9 seconds as a pass (see second screenshot). Is that a bug?

[patrickhulce]

@av3nger are you looking at the desktop numbers in PSI by chance?

What two Lighthouse settings are you comparing? :)

[av3nger]

@patrickhulce , both reports are for the desktop version. I'm trying to understand why PageSpeed Insights are highlighting the score in yellow, when it's a green?

I found this spreadsheet. And according to the spreadsheet, the 2.1 second Speed Index is between 99-100 score, which should make the coloring of the metric green. Same for the first meaningful paint metric.

PageSpeed Insights:

Chrome Litghthouse:

[patrickhulce]

The default scoring (including the spreadsheet) assumes mobile conditions. PSI adjusts the scoring for desktop according to the differences in connection and processing power.

Because there can be any mixture of throttling conditions while auditing a page in DevTools and the environment is not as predictable as PSI, the scoring rubric is not adjusted there which is why you are seeing the discrepancy. Essentially, assumptions about the comparability of the score are off when using any settings other than default throttling.

cc @paulirish @hwikyounglee I wonder if there should be some sort of visual indicator when Lighthouse was run with anything but the default settings that the score may be unexpected 🤔

[av3nger]

@patrickhulce, thank you for the explanation. This should be definitely explained much better. It's way too confusing at the moment - the spreadsheet is explaining the scores, you're thinking - ok, I just need 3,785 ms or less, and the speed index metric will be scored as "fast". Then you do a PSI scan, that is using Lighthouse, get the speed metric value of 2.1 seconds and you're labeled as "average"... 🤦‍♂️

[connorjclark]

at one point we considered showing the final-screenshot in the report, and wrapping that with the form factor (like a mobile device frame) as a way to indicate that the report is distinct from others. might apply here?

[paulirish]

mock from #9379

[connorjclark]

also considering showing the scoring curve in the metrics table somehow

not this exactly but the information we want to show is the same (the FCP mobile curve):