Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core(scoring): update TBT score curve #12576

Merged
merged 3 commits into from
Jun 1, 2021
Merged

core(scoring): update TBT score curve #12576

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4242df8
core(scoring): update TBT score curve
paulirish May 27, 2021
5ae62e0
update comment
paulirish May 27, 2021
949bcb9
test
paulirish May 27, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 14 additions & 7 deletions lighthouse-core/audits/metrics/total-blocking-time.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,14 +37,21 @@ class TotalBlockingTime extends Audit {
static get defaultOptions() {
return {
mobile: {
// According to a cluster telemetry run over top 10k sites on mobile, 5th percentile was 0ms,
// 25th percentile was 270ms and median was 895ms. These numbers include 404 pages. Picking
// thresholds according to our 25/75-th rule will be quite harsh scoring (a single 350ms task)
// after FCP will yield a score of .5. The following coefficients are semi-arbitrarily picked
// to give 600ms jank a score of .5 and 100ms jank a score of .999. We can tweak these numbers
// in the future. See https://www.desmos.com/calculator/bbsv8fedg5
// If determined from HTTP Archive data…
// SELECT
// APPROX_QUANTILES(tbt_value, 100)[OFFSET(8)] AS p08_tbt,
// APPROX_QUANTILES(tbt_value, 100)[OFFSET(25)] AS p25_tbt
// FROM (
// SELECT CAST(JSON_EXTRACT_SCALAR(report, '$.audits.total-blocking-time.numericValue') AS FLOAT64) AS tbt_value
// FROM `httparchive.lighthouse.2021_05_01_mobile`
// WHERE report is not NULL
// )
// …we'd use control points of 19 and 189, which leads to surprisingly harsh scoring.
//
// The following coefficients are semi-arbitrarily chosen, but start to approach the "correct" ones:
// See https://www.desmos.com/calculator/pwcgna1cvf go/lh8-tbt-curves
scoring: {
p10: 287,
p10: 200,
median: 600,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not drop the median down a bit too?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reducing it would exacerbate the effect of variance. This scoring curve has the biggest slope of all of the metrics, so I'm wary to increase the slope even more.

},
},
Expand Down
2 changes: 1 addition & 1 deletion lighthouse-core/test/audits/metrics/total-blocking-time-test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ describe('Performance: total-blocking-time audit', () => {

const outputMobile = await TBTAudit.audit(artifactsMobile, contextMobile);
expect(outputMobile.numericValue).toBeCloseTo(333, 1);
expect(outputMobile.score).toBe(0.85);
expect(outputMobile.score).toBe(0.75);
expect(outputMobile.displayValue).toBeDisplayString('330\xa0ms');

const artifactsDesktop = generateArtifacts({trace: lcpTrace,
Expand Down
2 changes: 1 addition & 1 deletion lighthouse-core/test/report/report-generator-test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ describe('ReportGenerator', () => {
expect(lines.length).toBeGreaterThan(100);
expect(lines.slice(0, 3).join('\n')).toMatchInlineSnapshot(`
"requestedUrl,finalUrl,category,name,title,type,score
\\"http://localhost:10200/dobetterweb/dbw_tester.html\\",\\"http://localhost:10200/dobetterweb/dbw_tester.html\\",\\"Performance\\",\\"performance-score\\",\\"Overall Performance Category Score\\",\\"numeric\\",\\"0.6\\"
\\"http://localhost:10200/dobetterweb/dbw_tester.html\\",\\"http://localhost:10200/dobetterweb/dbw_tester.html\\",\\"Performance\\",\\"performance-score\\",\\"Overall Performance Category Score\\",\\"numeric\\",\\"0.59\\"
\\"http://localhost:10200/dobetterweb/dbw_tester.html\\",\\"http://localhost:10200/dobetterweb/dbw_tester.html\\",\\"Performance\\",\\"first-contentful-paint\\",\\"First Contentful Paint\\",\\"numeric\\",\\"0.24\\"
"
`);
Expand Down
4 changes: 2 additions & 2 deletions lighthouse-core/test/results/sample_v2.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,7 +185,7 @@
"id": "total-blocking-time",
"title": "Total Blocking Time",
"description": "Sum of all time periods between FCP and Time to Interactive, when task length exceeded 50ms, expressed in milliseconds. [Learn more](https://web.dev/lighthouse-total-blocking-time/).",
"score": 1,
"score": 0.97,
"scoreDisplayMode": "numeric",
"numericValue": 116.79800000000023,
"numericUnit": "millisecond",
Expand Down Expand Up @@ -5276,7 +5276,7 @@
}
],
"id": "performance",
"score": 0.6
"score": 0.59
},
"accessibility": {
"title": "Accessibility",
Expand Down