Blocking js variability after LH 11.1

[cplprince]

FAQ

• Yes, my issue is not about variability or throttling.
• Yes, my issue is not about a specific accessibility audit (file with axe-core instead).

URL

to-das.cz

What happened?

After the upgrade on LH 11.1, some of the pages we tested started to have big variability in blocking js. Nothing in our architecture has changed; the LH is the only difference. The weird thing is that on the tested page, there is a constant number of blocking js. We do observe this on many other pages we test.

This can be observed on the desktop and mobile.

What did you expect?

A constant number of blocking js.

What have you tried?

We have checked if anything other than the LH version has changed. Every URL is tested 5 times. We checked manually for blocking JS and it should be constant every time. (in some cases even higher than the number measured by the lighthouse. We run the tester in the docker on the AWS infra. Before every test, we do a benchmark and based on the result of the benchmark we set the throttling(only for mobile, we do not throttle the CPU on the desktop tests)(devtools throttling). We do throttle the network as well(as specified in the throttling docs of the lighthouse).

If we try it on the LH 11.0, it works like before.(the number of blocking js is wrong, but at least there is no extreme variability).

How were you running Lighthouse?

node

Lighthouse Version

11.1.0

Chrome Version

114.0.5735.133

Node Version

node:20.4.0-slim

OS

linux/amd64

Relevant log output

No response

[adamraine]

What does the graph represent? Is it the number of pages with TBT high enough to fail? Or are you referring to the bootup-time audit which looks at blocking JS specifically?

[cplprince]

The graph represents render-blocking-resources, "application/javascript".

I will give you links, straight to our graphs for this test, so you can see what is going on. The page is in Czech language but the graphs should be readable for you either way.

TBT is alright: https://app.pagespeed.cz/r/43ad4f76a083/pages?publicHash=15fdaacc5239&range=90#celkovy-cas-blokovani-js-(tbt)

here is the blocking JS: https://app.pagespeed.cz/r/43ad4f76a083/sizes?publicHash=15fdaacc5239&range=90#pocet-blokujicich-js

For more clear information on what am i referring to, i will give you these code snippets to look at:

bjs: getItemsByMimeType(lhr.audits['render-blocking-resources'].details.items, 'application/javascript').length,

const getItemsByMimeType = createGetItemsByMimeType(lhr.audits['network-requests'].details.items);

export const createGetItemsByMimeType = (map: Array<{ url: string; mimeType: string }>) => {
	const dict = map.reduce((dict, { url, mimeType }) => {
		dict[url] = mimeType;
		return dict;
	}, {} as Record<string, string>);
	return <Item extends { url: string }>(items: Array<Item>, mimeType: string) => items.filter((item) => dict[item.url] === mimeType);
};

[adamraine]

I am not able to reproduce this, my environment setup:

• Chrome: 114.0.5735.133
• Node: 20.4.0
• Lighthouse: 11.1.0
• OS: Debian GNU/Linux rodete x86_64

The command I am running:

npx lighthouse https://to-das.cz --output=json --quiet | jq '.audits["render-blocking-resources"].details.items | length'

This consistently outputs 4 in my environment with no variability whatsoever. Perhaps something in your environment is influencing the results but without an adequate repro there isn't much we can do unfortunately.

[cplprince]

We are now trying to verify the inconsistency with the other options for throttling(I'll keep you posted when we get some solid results).

Either way, the number of the blocking js is wrong. You say, that it outputs 4 for you, our tester outputs 0-6. If we check manually on the webpage, how many are there, we get a 9. That is a much bigger number than LH returns.

window.performance.getEntriesByType('resource').filter(({ renderBlockingStatus }) => renderBlockingStatus === 'blocking')

returns 11 entries, 9 of them are JS.

[adamraine]

There are some differences between how Chrome determines render blocking status and how Lighthouse determines it. For more context on this problem please see #2065.

[adamraine]

Closing for now, if you have more info feel free to share it