web.dev consistently errors on Network.getResponseBody for main content

[brendankenny]

• run node lighthouse-cli/index.js https://web.dev --only-categories best-practices --view
• get Properly defines charset – Error!

(also happens in DevTools and PSI)

Example report

The error is

Required MainDocumentContent gatherer encountered an error: Protocol error (Network.getResponseBody): No resource with given identifier found

and occurs here in driver when called from main-document-content.

It seems unusual that the main document content wouldn't be available for the backend. @robdodson does mention some unusual things done with the service worker and "an app shell that pulls in an index.json of the page content when you go to other pages. We essentially replace main content area with this new content" that could be causing trouble (maybe the request really didn't have any content?).

[rpong]

We are experiencing the same issue with the lighthouse npm package. leaving a comment to follow this thread.

[robdodson]

btw I noticed that our service worker was being too aggressive (GoogleChrome/web.dev#3089). @brendankenny I tried to use canary to audit our staging server (https://web-dev-staging.appspot.com/) since it doesn't install the sw, but got an error

[connorjclark]

@robdodson #10883

should get fixed for next chromium canary

[Vadorequest]

Running into this too for the first time today. I got scored 0 for "Best Practices" when analyzing this website: https://nrn-v2-mst-aptd-at-lcz-sty-c1-ah8bmgx1e.vercel.app/en

LH Report: https://lighthouse-dot-webdotdevsite.appspot.com//lh/html?url=https%3A%2F%2Fnrn-v2-mst-aptd-at-lcz-sty-c1-ah8bmgx1e.vercel.app%2Fen#best-practices

[thepassle]

We're also running into this problem at modern-web.dev:

It seems like disabling the service worker 'fixes' the problem. We dont do anything fancy with our service worker — we either load an index.html either from network or from cache, we dont do any transformations on the html or anything like that. I really fail to see the connection between the service worker and defining a charset in the html ☹️

[patrickhulce]

I really fail to see the connection between the service worker and defining a charset in the html ☹️

Certain service workers mean that Lighthouse cannot fetch the HTML document to check for a charset. Error! means a Lighthouse bug, not something the page needs to fix (see #9968)

[thepassle]

We debugged a little bit more, we use a code snippet to reload the page when a new service worker has taken over:

if("serviceWorker"in navigator){let e;navigator.serviceWorker.addEventListener("controllerchange",()=>{e||(e=!0,window.location.reload())})}

❌ With that piece of code:

✅ Without that piece of code:

It seems that the reload is causing the best practices to error, hope that helps 🙂

EDIT:

In case anybody wants to workaround this, this seems to fix it. Also skips an unnecessary reload the first time a user visits your app (even though its so fast you might not have noticed)

        async function handleUpdate() {
          if ("serviceWorker"in navigator) {
            let refreshing;

            const oldSw = (await navigator.serviceWorker.getRegistration())?.active?.state;

            navigator.serviceWorker.addEventListener('controllerchange', async () => {
              if (refreshing) return;

              const newSw = (await navigator.serviceWorker.getRegistration())?.active?.state;

              if(oldSw === 'activated' && newSw === 'activating') {
                refreshing = true;
                window.location.reload();
              }
            });
          }
        }

        handleUpdate();

[patrickhulce]

Ah thanks @thepassle that helps a lot! I think this is happening because Chrome might only keep the network resources from the current page load in the memory cache which is what Lighthouse relies on to get the response body of the root document. We can do a bit more digging but this is a great head start 👍

related #8984 #11027

[robdodson]

We use a similar snippet on web.dev

[wreszelewski]

I run into this issue and made some debugging. I confirmed what @patrickhulce suggested. The problem is in NetworkAnalyzer.findOptionalMainDocument:

if (finalURL) {
      // equalWithExcludedFragments is expensive, so check that the finalUrl starts with the request first
      const mainResource = records.find(request => finalURL.startsWith(request.url) &&
        URL.equalWithExcludedFragments(request.url, finalURL));
      if (mainResource) return mainResource;
      // TODO: beacon !mainResource to Sentry, https://github.com/GoogleChrome/lighthouse/issues/7041
}

When reload is performed records variable contains multiple entries for the same url and only the last one can be get from memory cache. I can't find findOptionalMainDocument on main branch, so is there a point of fixing it basing on current release?

EDIT:
I found that problem still persists in new method:
https://github.com/GoogleChrome/lighthouse/blob/main/core/lib/dependency-graph/simulator/network-analyzer.js#L439

[frankii91]

I have the same problem, it's not only visible on my website. I have a more extensive worker service. Which is only a transitional version anyway. Can anyone help me solve the coding problem or improve the worker code? My website https://i-meble.eu

[adamraine]

@frankii91 The issue is not reproducible on that site

[frankii91]

Strange, the problem occurs all the time, I use the plugin in Chrome

[adamraine]

The fix appears in version 11.1.0, PSI and the Chrome extension still use 11.0.0

[tunetheweb]

@adamraine got another report for this site: https://leverx.com/

Testing it locally in Chrome (on 11.3.0) the error still shows. Any ideas why?

[adamraine]

@tunetheweb looks like a different issue that is reproducible without Lighthouse. I'll open a new issue since this one is pretty old.