report: add perf score explanation and link #9355

brendankenny · 2019-07-11T20:37:25Z

fixes #8717 (or at least takes a first stab)

let's bikeshed! Adapted from @csabapalfi's suggestion in #8717 (comment)

Regular blue link color was way too distracting in there, so adapted the gray.

patrickhulce · 2019-07-11T20:39:10Z

Gah beat me to it! 😭

brendankenny · 2019-07-11T20:39:10Z

lighthouse-core/lib/i18n/locales/en-XL.json

  "lighthouse-core/lib/lh-error.js | internalChromeError": {
    "message": "Âń îńt̂ér̂ńâĺ Ĉh́r̂óm̂é êŕr̂ór̂ óĉćûŕr̂éd̂. Ṕl̂éâśê ŕêśt̂ár̂t́ Ĉh́r̂óm̂é âńd̂ t́r̂ý r̂é-r̂ún̂ńîńĝ Ĺîǵĥt́ĥóûśê."
  },
+  "lighthouse-core/lib/lh-error.js | missingRequiredArtifact": {
+    "message": "R̂éq̂úîŕêd́ {artifactName} ĝát̂h́êŕêŕ d̂íd̂ ńôt́ r̂ún̂."
+  },


I think these two were from a merge mismatch where en-XL was checked in but #9284 wasn't built on top of that commit, so these weren't updated. Shouldn't happen again (at least for any new PRs)

brendankenny · 2019-07-11T20:39:37Z

lighthouse-core/report/html/report-styles.css

@@ -591,6 +592,9 @@
  color: var(--color-gray-600);
  margin: var(--section-padding-vertical) 0;
 }
+.lh-metrics__disclaimer a {
+  color: var(--color-gray-700);


800 was too dark compared to the text color

patrickhulce

Obviously love this, the only thing really different in my take was that I added explicit little subclauses to opportunities and diagnostics that they don't affect the score, just to really hammer it in :)

brendankenny · 2019-07-11T20:43:46Z

Obviously love this, the only thing really different in my take was that I added explicit little subclauses to opportunities and diagnostics that they don't affect the score, just to really hammer it in :)

lol I was literally just writing this comment:

We also know from experience that people don't read all the explanatory text, so maybe we also want to augment the opportunity and diagnostic descriptions so it's clear wherever users are looking? We could do something like

These optimizations can speed up your page load and may improve the performance score.

or go even more explicit

These optimizations are not a direct part of the performance score calculation, but can speed up your page load and improve the score.

or something

patrickhulce · 2019-07-11T20:49:26Z

These optimizations are not a direct part of the performance score calculation, but can speed up your page load and improve the score.

👍 I had it flipped in my branch (These optimizations can speed up your page load but do not directly affect your performance score.) but I like this a lot too :)

exterkamp

LGTM, nice addition to clear up confusion! I didn't realize it was a link at first, but I think it's great that the text links directly to the source!

exterkamp · 2019-07-11T21:59:14Z

These optimizations are not a direct part of the performance score calculation, but can speed up your page load and improve the score.

"These are generalized web performance optimizations, they do not directly contribute to the Lighthouse Performance score."

Just ideas 🤔

brendankenny · 2019-07-15T18:59:13Z

So trying out these variations:

Opportunities - These optimizations can speed up your page load but do not directly affect your performance score.

Opportunities - These optimizations can speed up your page load; they do not directly contribute to the Performance score.

express what we mean, but they also take up a (relatively) lot of visual space to explain what the opportunities are not, which is UI noise and it kind of undercuts the perceived usefulness of the opportunities right before we ask users to pay attention to them :)

Another option (kind of mentioned above) is phrasing positively, though that makes the score disclaimer more implicit. Something like:

Opportunities - These optimizations can speed up your page load, improving the scored performance metrics.

So maybe today (so we can ship 5.2), pick one of:

pick one of the above if you feel it really captures things
propose a variation of one of the above if you feel it's close but not quite there
propose something completely different if you have a flash of insight right now
let's punt on the opps/diagnostic string change

patrickhulce · 2019-07-15T19:27:53Z

Opportunities - These optimizations can speed up your page load; they do not directly contribute to the Performance score.

👍👍 I'm also fine with the version with 'but' in there. I think some version of this should be shipped.

Opportunities - These optimizations can speed up your page load, improving the scored performance metrics.

I think this moves us in the wrong direction. This makes it sounds like the opportunities are a direct factor of the score to me.

I'd be disappointed if we punted. This is one of the few things I tell pretty much every single person I introduce lighthouse to and, IMO, is central to reacting appropriately to the opportunities (treating them as a guide but not freaking out over a single resource showing up there you can't control).

brendankenny · 2019-07-15T20:12:01Z

Opportunities - These optimizations can speed up your page load; they do not directly contribute to the Performance score.

👍👍 I'm also fine with the version with 'but' in there. I think some version of this should be shipped.

so my issues are

half of it is telling you what opportunities can't do, which makes them seem less useful
the half that tells you what they can do is super vague
we say these can speed up page load and we say metrics make up the performance score, but we don't make explicit why you should care about opportunities then: these were chosen to help you improve the metrics.

Opportunities - These optimizations can speed up your page load, improving the scored performance metrics.

I think this moves us in the wrong direction. This makes it sounds like the opportunities are a direct factor of the score to me.

I can kind of see that, but is it a matter of phrasing or is it fundamental to the approach? :)

I'd be disappointed if we punted. This is one of the few things I tell pretty much every single person I introduce lighthouse to and, IMO, is central to reacting appropriately to the opportunities (treating them as a guide but not freaking out over a single resource showing up there you can't control).

totally agree on all this, I meant ship the first part because everyone seems fine enough on it and so we have something and then we can reconvene on the second part.

brendankenny · 2019-07-15T20:18:41Z

Opportunities - These optimizations can speed up your page load; they do not directly contribute to the Performance score.

👍👍 I'm also fine with the version with 'but' in there. I think some version of this should be shipped.

so my issues are

half of it is telling you what opportunities can't do, which makes them seem less useful

the half that tells you what they can do is super vague

we say these can speed up page load and we say metrics make up the performance score, but we don't make explicit why you should care about opportunities then: these were chosen to help you improve the metrics.

example report with this string to see it in context. Maybe the negative thing is just me :)

https://pumped-dash.glitch.me/

connorjclark · 2019-07-15T20:39:03Z

what about something under Metrics ala: Performance score is based only on these values, then you could drop the negative statement in Opportunities.

brendankenny · 2019-07-15T20:52:03Z

what about something under Metrics ala: Performance score is based only on these values, then you could drop the negative statement in Opportunities.

that's in there :)

See #9355 (comment) for why we might want more

meggynw · 2019-07-15T21:30:27Z

Hi all! Elizabeth forwarded this thread to me to see if I could weigh in (for those who don't know me, I'm a UXW for Chrome – hello!).

I like the general consensus where everyone is heading: we're offering useful insight while transparently mitigating user expectations. With the currently proposed drafts, one small nudge is that we typically put the positive factors first; so, it should be ordered from 'speed up' → 'score impact': "These optimizations can help your page load faster, but they don't directly impact the Performance score."

To address Brendan's feedback directly:

"half of it is telling you what opportunities can't do, which makes them seem less useful" ⇒ This is true, but mitigating expectations is valuable for user trust! It's much worse for someone to stop trusting Lighthouse because they spent tons of time on fixing the list of opportunities and then they didn't see any improvement to their score
"the half that tells you what they can do is super vague" ⇒ I agree! Can we be more explicit and call out exactly which metrics are most likely to be positively impacted?
"we say these can speed up page load and we say metrics make up the performance score, but we don't make explicit why you should care about opportunities then: these were chosen to help you improve the metrics" ⇒ From a string perspective, this looks like a fundamental contradiction: these actions will not improve your score, but they will improve your metrics? Any chance you can flesh that out so I can better help out?

patrickhulce · 2019-07-15T22:18:28Z

Thanks for the valuable insight @meggynw!!

Can we be more explicit and call out exactly which metrics are most likely to be positively impacted?

The primary reason it's so vague is that each opportunity affects different aspects of load performance. Some will affect "FCP", others will affect "TTI", still others will not directly reflect in our set of metrics but still positively impact the amount of time taken for the entire page to download. We had thrown around in years past the idea of directly stating what each opportunity does, but it's certainly a UX challenge for information overload and more on this in the next response...

this looks like a fundamental contraction: these actions will not improve your score, but they will improve your metrics? Any chance you can flesh that out so I can better help out?

One of the main challenges with the opportunities is that they can impact the page in many different ways. While we are definitely prioritizing them by their estimated overall impact (and hiding them completely if we think they're not worthwhile), it's not an exact science and promising an exact score increase would be overpromising (and as you so excellently put it, jeopardize user trust).

Another mismatch with the opportunities:score relationship is that we construct our incentives around the score while opportunities are always meant to be just helpful advice. Example: if you're doing really well on a particular metric, we might give you a great score there to incentivize moving on to another metric, but there still might be room for improvement we found, so an opportunity could still be visible (albeit low priority). On the flip side, there are cases where the score is very low but there are few automatically identifiable opportunities for improvement. Disconnecting the score from the opportunities goes a long way IMO to helping users better interpret the information we're giving.

meggynw · 2019-07-17T18:10:02Z

Thanks for the thorough answer :) That all makes a lot of sense.

Based on my understanding, I've tweaked our string to the following. I propose "suggestions" instead of "optimizations" as it makes the noun is less redundant with "page load faster"; it also emphasizes that these are nice-to-haves instead of priority items. I've tried to use "directly related" in the 2nd sentence to clarify that, while there may be a minor positive impact, devs can't rely on opportunities for score improvements.

"These suggestions can help your page load faster. They aren't directly related to Performance metrics or the total score."

Reactions? Open to feedback :)

patrickhulce · 2019-07-17T21:26:51Z

"These suggestions can help your page load faster. They aren't directly related to Performance metrics or the total score."

Reactions! :)

I like "suggestions" here 👍
I don't love saying they're not related to "performance metrics" at large. All of the opportunities are computed on some performance metric, is just might not help your score due to our weighting and score curves.

My tweak might be:

These suggestions can help your page load faster. They do not directly affect the overall performance score.

I personally like "affect" compared to "relate" since I think it gets at the heart of people's fears that the opportunity result is negatively affecting their score.

meggynw · 2019-07-17T21:32:03Z

That string LGTM! I'd just use the contraction "don't" for a more conversational tone. Also, another alternative to "affect" is "impact," but I don't have a strong preference.

…

On Wed, Jul 17, 2019 at 2:27 PM Patrick Hulce ***@***.***> wrote: "These suggestions can help your page load faster. They aren't directly related to Performance metrics or the total score." Reactions! :) - I like "suggestions" here 👍 - I don't love saying they're not related to "performance metrics" at large. All of the opportunities are computed on *some* performance metric, is just might not help your score due to our weighting and score curves. My tweak might be: These suggestions can help your page load faster. They do not directly affect the overall performance score. I personally like "affect" compared to "relate" since I think it gets at the heart of people's fears that the opportunity result is negatively affecting their score. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9355?email_source=notifications&email_token=ALEGLLMBURWHY5T2IMJNQQ3P76FDNA5CNFSM4IBUHTYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2GUNOI#issuecomment-512575161>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALEGLLOX7L5FBFKFNVE7G63P76FDNANCNFSM4IBUHTYA> .

-- Meggyn Watkins | UX Writer | Chrome | meggyn@ <meggyn@google.com>

brendankenny · 2019-07-18T06:21:29Z

That string LGTM! I'd just use the contraction "don't" for a more
conversational tone. Also, another alternative to "affect" is "impact," but
I don't have a strong preference.

I think one nuance that was lost in the end here was that Opportunities actually are Lighthouse's best current advice for how users can impact their score. The distinction is hard to communicate :)

A lot of users interpret it as "my score is bad because I'm not doing the things in this list", when really it's "your score is bad because the metrics showed the page loaded slowly; here's a list of things we think should (but won't necessarily) improve the page load and thus the metrics and thus your score".

They can do everything in that list and still have a slow page (the opportunities and the metrics aren't exhaustive), or they could not do everything in the list and have a fast page (if you have a 2s opportunity on a 1s page load that's a bug but some form of this does happen).

Anyway, the fact that we don't add up the opportunities to make a score is what we'd like to clear up. If we land too far on the conservative side ("here is some random trivia about your page that will never ever have anything to do with your score" :), that's probably ok for a while.

paulirish

on calling out optimizations+scoring in the report:

I think it's unnecessary.

I like the new "disclaimer" text for metrics and the long-awaited link to scoring docs. The prominent link directs people to a page that should communicate clearly that about the relationship of opportunities results & score.

We can start with adding just that metrics disclaimer sentence along with refreshed scoring docs.

paulirish · 2019-07-18T17:37:43Z

lighthouse-core/report/html/renderer/util.js

@@ -557,7 +557,7 @@ Util.numberDateLocale = 'en';
 */
 Util.UIStrings = {
  /** Disclaimer shown to users below the metric values (First Contentful Paint, Time to Interactive, etc) to warn them that the numbers they see will likely change slightly the next time they run Lighthouse. */
-  varianceDisclaimer: 'Values are estimated and may vary.',
+  varianceDisclaimer: 'Values are estimated and may vary. The performance score is [based only on these metrics](https://github.com/GoogleChrome/lighthouse/blob/master/docs/scoring.md#how-are-the-scores-weighted).',


@egsweeny do you we can should to the scoring doc in github or to https://developers.google.com/web/tools/lighthouse/v3/scoring ... in reality, both need updating, but google/WebFundamentals#7724 is already started.

I kinda prefer the devsite page.. it's SEO is a bit stronger. And looks nice. :)

brendankenny · 2019-07-18T20:10:41Z

on calling out optimizations+scoring in the report:

I think it's unnecessary.

I like the new "disclaimer" text for metrics and the long-awaited link to scoring docs. The prominent link directs people to a page that should communicate clearly that about the relationship of opportunities results & score.

We can start with adding just that metrics disclaimer sentence along with refreshed scoring docs.

Trying to represent @exterkamp and myself:

we should just go with the (perhaps too) strongly worded warning. In an ideal world we could do just the metrics warning change and see if confusion drops, but our feedback routes for this are too tenuous and hard to quantify. If confusion continues, a likely outcome is that we'll just continue to debate the few signals we get from it and do nothing about it until the next UI change.

Instead, it doesn't do much harm to go overboard on waving people off opportunities, and the fact that we will be seeing it in the report every day means that if we do find it doesn't express the situation well we'll be more likely to revise in the future.

brendankenny · 2019-07-18T20:13:20Z

(though for the record I still 100% agree with everything I wrote above :)

brendankenny · 2019-07-18T20:22:33Z

@paulirish sounds like he can live with this if directly affect is a link to an entry in the scoring docs explaining exactly what that means :)

brendankenny · 2019-07-18T20:37:58Z

updated with strings. Links are in progress (going to link to update from google/WebFundamentals#7724)

patrickhulce

linking directly affect SGTM! :)

great compromises team!

patrickhulce · 2019-07-18T22:38:22Z

lighthouse-core/config/default-config.js

-  /** Description of the opportunity section of the Performance category. 'Optimizations' could also be 'recommendations' or 'suggestions'. Within this section are audits with imperative titles that suggest actions the user can take to improve the loading performance of their web page. */
-  loadOpportunitiesGroupDescription: 'These optimizations can speed up your page load.',
+  /** Description of the opportunity section of the Performance category. 'Suggestions' could also be 'recommendations'. Within this section are audits with imperative titles that suggest actions the user can take to improve the loading performance of their web page. */
+  loadOpportunitiesGroupDescription: 'These suggestions can help your page load faster. They don\'t [directly affect](https://github.com/GoogleChrome/lighthouse/blob/master/docs/scoring.md#how-are-the-scores-weighted) the Performance score.',


~~maybe we should be sha (or at least version) tagging these links if they're going to live in a report forever?~~

ignore this I just read the "links are in progress" for the web fundamentals update :)

~~maybe we should be sha (or at least version) tagging these links if they're going to live in a report forever?~~

ignore this I just read the "links are in progress" for the web fundamentals update :)

actually, maybe we won't wait for that. At worst, 5.2 users will get a special easter egg doc link. Added the hash as suggested, though :)

report: add perf score explanation and link

1486758

brendankenny requested review from exterkamp and paulirish as code owners July 11, 2019 20:37

googlebot added the cla: yes label Jul 11, 2019

brendankenny commented Jul 11, 2019

View reviewed changes

patrickhulce reviewed Jul 11, 2019

View reviewed changes

lint and link assertion

85c2b36

vercel bot deployed to staging July 11, 2019 20:49 View deployment

paulirish added the 5.2 label Jul 11, 2019

exterkamp approved these changes Jul 11, 2019

View reviewed changes

csabapalfi approved these changes Jul 12, 2019

View reviewed changes

brendankenny mentioned this pull request Jul 18, 2019

misc: fix yarn.lock and translations master drift #9404

Merged

paulirish reviewed Jul 18, 2019

View reviewed changes

opps and diags

a9f32ea

vercel bot deployed to staging July 18, 2019 20:36 View deployment

patrickhulce reviewed Jul 18, 2019

View reviewed changes

cli snapshot

6b64738

vercel bot deployed to staging July 18, 2019 22:49 View deployment

use md score guide for now

ca08dfe

vercel bot deployed to staging July 19, 2019 23:55 View deployment

brendankenny merged commit 8eb0cde into master Jul 20, 2019

brendankenny deleted the scoreexp branch July 20, 2019 00:36

paulirish pushed a commit that referenced this pull request Nov 6, 2019

report: add link to docs on performance scoring (#9355)

85ae6ea

snyk-bot mentioned this pull request Mar 21, 2020

[Snyk] Upgrade lighthouse from 5.1.0 to 5.6.0 godaddy/lighthouse4u#13

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

report: add perf score explanation and link #9355

report: add perf score explanation and link #9355

brendankenny commented Jul 11, 2019

patrickhulce commented Jul 11, 2019

brendankenny Jul 11, 2019

brendankenny Jul 11, 2019

patrickhulce left a comment

brendankenny commented Jul 11, 2019

patrickhulce commented Jul 11, 2019

exterkamp left a comment

exterkamp commented Jul 11, 2019

brendankenny commented Jul 15, 2019 •

edited

Loading

patrickhulce commented Jul 15, 2019

brendankenny commented Jul 15, 2019 •

edited

Loading

brendankenny commented Jul 15, 2019

connorjclark commented Jul 15, 2019

brendankenny commented Jul 15, 2019

meggynw commented Jul 15, 2019 •

edited

Loading

patrickhulce commented Jul 15, 2019

meggynw commented Jul 17, 2019

patrickhulce commented Jul 17, 2019

meggynw commented Jul 17, 2019 via email

brendankenny commented Jul 18, 2019

paulirish left a comment

paulirish Jul 18, 2019

brendankenny commented Jul 18, 2019

brendankenny commented Jul 18, 2019

brendankenny commented Jul 18, 2019

brendankenny commented Jul 18, 2019

patrickhulce left a comment

patrickhulce Jul 18, 2019 •

edited

Loading

brendankenny Jul 19, 2019

report: add perf score explanation and link #9355

report: add perf score explanation and link #9355

Conversation

brendankenny commented Jul 11, 2019

patrickhulce commented Jul 11, 2019

brendankenny Jul 11, 2019

Choose a reason for hiding this comment

brendankenny Jul 11, 2019

Choose a reason for hiding this comment

patrickhulce left a comment

Choose a reason for hiding this comment

brendankenny commented Jul 11, 2019

patrickhulce commented Jul 11, 2019

exterkamp left a comment

Choose a reason for hiding this comment

exterkamp commented Jul 11, 2019

brendankenny commented Jul 15, 2019 • edited Loading

patrickhulce commented Jul 15, 2019

brendankenny commented Jul 15, 2019 • edited Loading

brendankenny commented Jul 15, 2019

connorjclark commented Jul 15, 2019

brendankenny commented Jul 15, 2019

meggynw commented Jul 15, 2019 • edited Loading

patrickhulce commented Jul 15, 2019

meggynw commented Jul 17, 2019

patrickhulce commented Jul 17, 2019

meggynw commented Jul 17, 2019 via email

brendankenny commented Jul 18, 2019

paulirish left a comment

Choose a reason for hiding this comment

paulirish Jul 18, 2019

Choose a reason for hiding this comment

brendankenny commented Jul 18, 2019

brendankenny commented Jul 18, 2019

brendankenny commented Jul 18, 2019

brendankenny commented Jul 18, 2019

patrickhulce left a comment

Choose a reason for hiding this comment

patrickhulce Jul 18, 2019 • edited Loading

Choose a reason for hiding this comment

brendankenny Jul 19, 2019

Choose a reason for hiding this comment

brendankenny commented Jul 15, 2019 •

edited

Loading

brendankenny commented Jul 15, 2019 •

edited

Loading

meggynw commented Jul 15, 2019 •

edited

Loading

patrickhulce Jul 18, 2019 •

edited

Loading