Define link processing for prefetch

[noamr]

Prefetch is simply a fetch without post-processing the resource, and with a special header: Sec-Purpose: Prefetch, which would be added in the Fetch-Metadata spec. See that issue for use-cases for the special header.

Closes #5229
Closes w3c/resource-hints#86
Closes w3c/resource-hints#74
Closes whatwg/fetch#1008

• At least two implementers are interested (and none opposed):

  • Support from Mozilla & Chromium.
  • Waiting to align with Apple re. Sec-Purpose: Header

• Tests are written and can be reviewed and commented upon at:

  • Add multiple tests for prefetch behavior web-platform-tests/wpt#35707
  • CSP prefetch-src is already tested

• Implementation bugs are filed:

  • Chrome:
    1345207: remove the five minute rule
    1358419: Sec-Purpose header
    626081: issues with Accept
    801561: Prefetch src
    1352371: Remove as=document special treatment
  • Firefox:
    1457204: CSP prefetch-src
    1788167: Align prefetch with spec
  • Safari: https://bugs.webkit.org/show_bug.cgi?id=49238

(See WHATWG Working Mode: Changes for more details.)

---

/infrastructure.html ( diff )
/links.html ( diff )

[domenic]

Well, this is certainly simple; I have no qualms with the algorithm. However:

Already implemented

seems unlikely? My understanding is there's a lot of non-interop on prefetch from various implementers, and nobody currently implements this proposal exactly. Or maybe Firefox does, and that's why they're N/A for the bug link?

The Safari bug seems to be about something that isn't captured in this spec proposal. If we don't want to allow prefetch during loading, that should be normatively enforced.

The Chrome bug seems to be about removing the 5-minute rule, but are there more things Chrome does, e.g. use a memory cache, special treatment of other headers besides max-age, etc. which also need bugs?

[domenic]

This is a bit surprising to me. Tests would be good!

[noamr]

Working on it :)

[domenic]

Should we modify https://fetch.spec.whatwg.org/#concept-request-priority somehow?

[noamr]

I don't think so. It's still implementation-defined.

[domenic]

As noted in #5229 I guess we should add some headers? But which ones? See https://wicg.github.io/nav-speculation/prefetch.html#prefetch-algorithms step 5 for what nav-speculation is doing. It includes a carveout for vendor-specific headers which I dislike but decided was OK back in WICG/nav-speculation#133 (comment) .

[noamr]

We already send Purpose: Prefetch. I think we should also send Sec-Purpose: Prefetch and have it as part of the spec.

[domenic]

Should we allow/encourage/require Purpose: Prefetch? What about X-moz: prefetch or X-Purpose: preview? Those latter two values were tested, I think, for things besides <link rel="prefetch">; do you know what Firefox and Safari do for <link rel="prefetch">?

[noamr]

I think currently we're specifying as Sec-Purpose: Prefetch, and everything else is proprietary and UAs will probably keep it for backwards compat.

[annevk]

I would hope they plan on removing the non-compliant headers. If the plan is that everyone ends up with multiple incompatible headers that does not seem great and at that point there's no real value in using Sec- either, right?

[noamr]

Well, this is certainly simple; I have no qualms with the algorithm. However:

Already implemented

seems unlikely? My understanding is there's a lot of non-interop on prefetch from various implementers, and nobody currently implements this proposal exactly. Or maybe Firefox does, and that's why they're N/A for the bug link?

Actually Safari implements this exactly. Firefox have an intricate way to de-prioritize prefetches, i.e. performing them one by one and only after the window onload event is fired. I wonder if this needs to be interoperable, or if this can be considered as their implementation of determining request priority.

The Safari bug seems to be about something that isn't captured in this spec proposal. If we don't want to allow prefetch during loading, that should be normatively enforced.

I referenced this bug because the bug mentions that prefetch is not entirely spec'ed yet.
I'm OK with discussing adding the "only before onload" bit to the spec but I thought it would be good to have a common denominator first and then modify it if there's consensus.

The Chrome bug seems to be about removing the 5-minute rule, but are there more things Chrome does, e.g. use a memory cache, special treatment of other headers besides max-age, etc. which also need bugs?

From what I understood, the special treatment is for navigations, which would be ported to nav-speculation in due time. But I'll double check. Regarding using the memory cache, it feels more of an optimization, I'm not sure it's observable in any way.

[domenic]

Actually Safari implements this exactly.

OK, so they should probably wontfix https://bugs.webkit.org/show_bug.cgi?id=49238 then?

Firefox have an intricate way to de-prioritize prefetches, i.e. performing them one by one and only after the window onload event is fired. I wonder if this needs to be interoperable, or if this can be considered as their implementation of determining request priority.

It feels like a bit of a stretch, but, maybe it's OK, given how vague priority is.

My main concern is that if there are patterns that browsers should follow to conform to websites' expectations and usage, then we should try to encode that, so that browsers implementing the spec naively don't end up causing sites to perform poorly in their browser. It's not clear to me whether or not that's the case.

Regarding using the memory cache, it feels more of an optimization, I'm not sure it's observable in any way.

My understanding was that the memory cache prevents future fetches from hitting the network and/or service worker, which would otherwise do so. I might be wrong.

---

Also a crbug comment from 2013 indicates that whether or not <link rel="prefetch"> blocks the window load event is not interoperable. It looks like in this spec PR we retain the default which is that it does block the load event. That would benefit from tests, and if there is cross-browser disagreement, discussion.

[noamr]

Actually Safari implements this exactly.

OK, so they should probably wontfix https://bugs.webkit.org/show_bug.cgi?id=49238 then?

I think they should consider how they implement fetching priority if it bothers their users. But perhaps it's not a spec concern.

Firefox have an intricate way to de-prioritize prefetches, i.e. performing them one by one and only after the window onload event is fired. I wonder if this needs to be interoperable, or if this can be considered as their implementation of determining request priority.

It feels like a bit of a stretch, but, maybe it's OK, given how vague priority is.

My main concern is that if there are patterns that browsers should follow to conform to websites' expectations and usage, then we should try to encode that, so that browsers implementing the spec naively don't end up causing sites to perform poorly in their browser. It's not clear to me whether or not that's the case.

I agree with that. I wonder if this can be done incrementally.

Regarding using the memory cache, it feels more of an optimization, I'm not sure it's observable in any way.

My understanding was that the memory cache prevents future fetches from hitting the network and/or service worker, which would otherwise do so. I might be wrong.

Right, so a test can be written that checks if consuming a disk-cachable prefetch has hit a service worker. Will give it a go.

Also a crbug comment from 2013 indicates that whether or not <link rel="prefetch"> blocks the window load event is not interoperable. It looks like in this spec PR we retain the default which is that it does block the load event. That would benefit from tests, and if there is cross-browser disagreement, discussion.

I believe it should definitely not block the load events. In Firefox it wouldn't even take place before the load event. I will check what WPTs tell us.

[noamr]

My understanding was that the memory cache prevents future fetches from hitting the network and/or service worker, which would otherwise do so. I might be wrong.

Right, so a test can be written that checks if consuming a disk-cachable prefetch has hit a service worker. Will give it a go.

I checked this. The prefetch cache in chrome is actually some special consideration about when to invalidate, but it's really the regular disk cache. It doesn't prevent SWs from running again on the same resource.

[domenic]

#5229 (comment) discusses even more departures that Chrome has from this proposed spec: some special behavior for as=document that allows it to be used to speed up cross-site navigations. I am a bit frustrated that was not mentioned either in the implementer support section. Are we planning to remove that in Chrome? If not then I think it would be bad to move forward with this spec proposal which specifies something so different...

[noamr]

#5229 (comment) discusses even more departures that Chrome has from this proposed spec: some special behavior for as=document that allows it to be used to speed up cross-site navigations. I am a bit frustrated that was not mentioned either in the implementer support section. Are we planning to remove that in Chrome? If not then I think it would be bad to move forward with this spec proposal which specifies something so different...

Yes, the idea as I understood it was to move the as=document things to nav-speculation. I didn't mention it because I wanted to specify the common-denominator first, and then discuss the "chrome only" bits and see what of it needs to stay in . What frustrates me is that it's terribly difficult to get people to join this conversation over something that's implemented differently in each browser and not specified... Could use some help with that.

[domenic]

Well, it seems like we might have a good basis? If Safari matches this PR, and Firefox maybe matches it except for some stuff about load timing which we think is OK, then Chrome is the odd one out, right? We don't really need that much of a conversation; we just need Chrome to commit to matching the other two browsers.

To be clear, I don't think interpreting Chrome's current implementation as a superset of this PR is a great application of the WHATWG working mode. "Yes, we implement this PR but also a bunch of other stuff triggered by the same link rel" would justify a lot of weirdness, if we extended that rule more broadly.

We could just merge this on the basis of Firefox + Safari support, saying we don't know Chrome's position on it, but (a) that's a weird thing for two Chrome engineers to conclude 🙂 and (b) if there's a chance that Chrome wants to actually steer the semantics of rel=prefetch toward something other than what's in this PR in the future, then it feels like this PR is a bit deceptive, especially for Firefox + Safari engineers who might think it indicates an agreement on a specific behavior, when in reality it doesn't.

[noamr]

Well, it seems like we might have a good basis? If Safari matches this PR, and Firefox maybe matches it except for some stuff about load timing which we think is OK, then Chrome is the odd one out, right? We don't really need that much of a conversation; we just need Chrome to commit to matching the other two browsers.

To be clear, I don't think interpreting Chrome's current implementation as a superset of this PR is a great application of the WHATWG working mode. "Yes, we implement this PR but also a bunch of other stuff triggered by the same link rel" would justify a lot of weirdness, if we extended that rule more broadly.

We could just merge this on the basis of Firefox + Safari support, saying we don't know Chrome's position on it, but (a) that's a weird thing for two Chrome engineers to conclude 🙂 and (b) if there's a chance that Chrome wants to actually steer the semantics of rel=prefetch toward something other than what's in this PR in the future, then it feels like this PR is a bit deceptive, especially for Firefox + Safari engineers who might think it indicates an agreement on a specific behavior, when in reality it doesn't.

Update: restrictive prefetch (<link rel=prefetch as=document>) is used (almost) exclusively by Google search prefetching signed exchanges (a great use case!) and it seems like speculation rules could be a better supported future there. So I suggest we continue down the road of this patch, and work separately on sunsetting restrictive prefetch, which is somewhat an extension of the prefetch feature anyway and not the core of it. WDYT @domenic?

[domenic]

That sounds good. Here are some things we should be sure to test:

• No special treatment of as=document
• No 5 minute rule (cache headers must be respected)
• Does not block the load event
• Whether any additional headers (Sec-Purpose, Purpose, X-moz, X-Purpose) are sent (depending on what we put in the spec)
• Whether Accept is left as its default value. (I think I saw a recent Chromium bug on this.)
• Is blocked by CSP's prefetch-src but not by other CSP things (e.g., frame-src and img-src do not block requests for HTML documents or images)
• Always uses prefetch destination, and ignores as="" (both: as seen by service workers and/or Sec-Fetch-Dest; and, if you use as="img" you can still consume the result inside a fetch() or whatever.)
• Storage partitioning is applied, including for documents

Once we have those test results, we can know exactly how far everyone is from the proposal, and confirm that everyone is on board with making the necessary changes.

[noamr]

• Whether Accept is left as its default value. (I think I saw a recent Chromium bug on this.)

https://bugs.chromium.org/p/chromium/issues/detail?id=1357407
This bug refers to a page which uses as=document :(

[noamr]

I dug deep into Accept today... I wonder if Vary: Accept makes it so that <link rel=prefetch> is more of a footgun than something useful. You can think that you're prefetching a subresource for subsequent pages, and then someone in the server adds Vary: Accept and you're left with a double request, in a worst place from where you started.

I propose the following:

• When prefetching, send the same Accept header as when navigating from the URL address bar
• When consuming a prefetch, ignore Accept: Vary matching.

[domenic]

• When prefetching, send the same Accept header as when navigating from the URL address bar

In spec land, I think this is the same as when fetching with fetch() or <img src="">. What is your intent with this phrasing? To match that default, or to do something different, more like a navigation? See also whatwg/fetch#274 .

(To be clear, we don't need to solve that entire issue before proceeding here, but I want to get a sense of what the intent is, and maybe encode it in the spec and tests if we can do so despite the shaky foundations.)

• When consuming a prefetch, ignore Accept: Vary matching.

(Assuming you mean Vary: Accept.)

"Consuming a prefetch" just means "making a fetch", right? A fetch that will hit the HTTP cache that the prefetch might have populated?

I'm not sure it's a good idea to start selectively ignoring Vary depending on who initiated the fetch. Is there precedent for that elsewhere in the platform?

If there is significant evidence of Vary: Accept on <link rel=prefetch> causing problems in the wild, then maybe this is worth doing, but my instinct is to treat Vary for prefetch fetches the same as it is treated for others.

[noamr]

• When prefetching, send the same Accept header as when navigating from the URL address bar

In spec land, I think this is the same as when fetching with fetch() or <img src="">. What is your intent with this phrasing? To match that default, or to do something different, more like a navigation? See also whatwg/fetch#274 .

Not sure yet, still investigating. It needs to tell the server which mime types the user agent supports and do some intricate content negotation thingies like SXG quality, and OTOH be ignored by the client HTTP cache.

(To be clear, we don't need to solve that entire issue before proceeding here, but I want to get a sense of what the intent is, and maybe encode it in the spec and tests if we can do so despite the shaky foundations.)

• When consuming a prefetch, ignore Accept: Vary matching.

(Assuming you mean Vary: Accept.)

Yes, of course :)

"Consuming a prefetch" just means "making a fetch", right? A fetch that will hit the HTTP cache that the prefetch might have populated?

Yes.

I'm not sure it's a good idea to start selectively ignoring Vary depending on who initiated the fetch. Is there precedent for that elsewhere in the platform?

Well it's a subset of Chromium's current "5 minute rule" for prefetches, which would be the precedent.
Also the preload cache is somewhat of a precedent.
I think the Vary: Accept thing is one reason why removing this rule might cause a regression.
But IMO regardless of precedent it makes perfect sense for prefetch - Vary: Accept is affected by two things - the type of resource and which browser this is. In this case the HTTP cache knows that we're in the same browser, and with prefetch we didn't know the type of resource when we fetched.

I think the alternative to doing this would be to deprecate prefetch (or to keep something like the current 5 minute rule). It becomes too much a footgun where by changing the Vary header you can make prefetch from a performance benefit to a performance regression.

[noamr]

That sounds good. Here are some things we should be sure to test:

• No special treatment of as=document
• No 5 minute rule (cache headers must be respected)
• Does not block the load event
• Whether any additional headers (Sec-Purpose, Purpose, X-moz, X-Purpose) are sent (depending on what we put in the spec)
• Whether Accept is left as its default value. (I think I saw a recent Chromium bug on this.)
• Is blocked by CSP's prefetch-src but not by other CSP things (e.g., frame-src and img-src do not block requests for HTML documents or images)
• Always uses prefetch destination, and ignores as="" (both: as seen by service workers and/or Sec-Fetch-Dest; and, if you use as="img" you can still consume the result inside a fetch() or whatever.)
• Storage partitioning is applied, including for documents

Once we have those test results, we can know exactly how far everyone is from the proposal, and confirm that everyone is on board with making the necessary changes.

Done.

• Add multiple tests for prefetch behavior web-platform-tests/wpt#35707 for new tests
• CSP prefetch-src is already tested

Updated the OP.

Still need to figure out if what we do here with Accept is what we want, it's a straw man test for my proposal.

[dieulot]

If there is significant evidence of Vary: Accept on causing problems in the wild, then maybe this is worth doing, but my instinct is to treat Vary for prefetch fetches the same as it is treated for others.

When people use a library to prefetch HTML (which allows speeding one’s website very easily), like my library instant.page, or quicklink/Flying Pages/etc., they expect it to just work and generally don’t know jack about caching headers. Even if they know they don’t have those conflicts in their head. Caching conflicts with prefetch are always a bad surprise in general.

In the wild, Shopify uses Vary: Accept, causing problems. That’s a lot of sites where people just can’t set “correct” caching headers to let prefetch do its magic.

I think standardizing ignoring Vary: Accept would be a great way to ensure in the long term that various changes to browsers' Accept headers don’t make a collateral damage of prefetch.

[noamr]

• When prefetching, send the same Accept header as when navigating from the URL address bar

In spec land, I think this is the same as when fetching with fetch() or <img src="">. What is your intent with this phrasing? To match that default, or to do something different, more like a navigation? See also whatwg/fetch#274 .

It's actually the same as when fetching a document (iframe/window.open etc). You can prefetch anything that your browser can load. So I suggest to add prefetch to the switch statement in the fetch spec of the Accept header.

Looking at whatwg/fetch#274 that switch statement is anyway far from useful, but understanding now that prefetch should match whatever document does would help when we fix that.

[noamr]

@emilio @sefeng211 @bdekoz: Gecko thoughts?
@achristensen07: WebKit thoughts?

Trying to make prefetch interoperability less insane so please pitch in :)

[domenic]

Looking pretty solid, assuming we can get cross-browser agreement... the test results are somewhat grim though! https://github.com/web-platform-tests/wpt/pull/35707/checks?check_run_id=8191297010

[annevk]

So as I understand it this essentially makes prefetch a way to fill the HTTP cache, with some special request headers and optionally some priority adjustments.

(This means that service workers will still be consulted.)

Chatting with colleagues that seems like an acceptable model.

[annevk]

Two dots at the end here seems wrong?

[noamr]

Historically a big use case for <link rel=prefetch> was search results on google.com. And those are pretty much always cross-origin URLs. If that pattern was more widely adopted those would by definition all be wasted bytes as the partitions wouldn't ever be able to match up.

A detailed output:
https://docs.google.com/spreadsheets/d/1N7fHysHI-Yx1taBppcGN3Z_DK8Y--fc4XZmtesIrMp4/edit?usp=sharing

Prefetch is predominantly used for same-origin scripts (57%) and same-origin CSS (18%).
Cross-site HTML comes a distant fourth (after cross-site scripts, which would still work in most cases) at 3.5%, and the most common ones in that category are iframes such as cookie-consent providers, which would also still work.

My take on this data is that prefetch used to be a search-engine feature, and today it's more of a landing-page feature (you serve a very lean and fast landing page that prefetches the scripts & styles of the complex main page/web-app)

[noamr]

Hey @noamr, it would help if you documented the rationale behind Sec-Purpose in a note or some such.

Done in whatwg/fetch#1576