User-Agent Client Hints & UA Reduction

[miketaylr]

Hi there TAG,

I'm requesting a TAG review of User-Agent Client Hints and the plan to reduce the information available in the User-Agent header, and related JS APIs. The review in issue #467 was previously closed, but I’m requesting a new review now that Chrome is interested in pursuing this plan again.

User-Agent Client Hints defines a set of Client Hints that aim to provide developers with the ability to perform agent-based content negotiation when necessary, while avoiding the historical baggage and passive fingerprinting surface exposed by the venerable User-Agent header.

• Explainer: https://github.com/WICG/ua-client-hints#explainer-reducing-user-agent-granularity
• Specification URL: https://wicg.github.io/ua-client-hints/
• Tests:
  https://github.com/web-platform-tests/wpt/tree/master/client-hints
  https://github.com/web-platform-tests/wpt/tree/master/ua-client-hints
• Security and Privacy self-review: https://github.com/WICG/ua-client-hints/blob/main/tag-security-privacy.md
• GitHub repo (if you prefer feedback filed there): https://github.com/WICG/ua-client-hints
• Primary contacts (and their relationship to the specification):
  • Mike Taylor (@miketaylr - editor)
• Organization(s)/project(s) driving the specification: Chromium
• Key pieces of existing multi-stakeholder review or discussion of this specification:
  Mozilla’s position: https://mozilla.github.io/standards-positions/#ua-client-hints
  Early review of User-Agent information as Client Hints concept: Migrating some high-entropy HTTP request headers to Client Hints. #320 (comment)
  Review of “Partial freezing of the UA string”, which was closed when our plans to proceed were paused due to the pandemic: Partial freezing of the User-Agent string #467 (comment)
• External status/issue trackers for this specification (publicly visible, e.g. Chrome Status):
  https://chromestatus.com/feature/5995832180473856
  https://chromestatus.com/feature/5733498725859328

Further details:

• I have reviewed the TAG's Web Platform Design Principles
• The group where the work on this specification is currently being done: WICG
• The group where standardization of this work is intended to be done (if current group is a community group or other incubation venue): W3C
• Major unresolved issues with or opposition to this specification: There's some opposition recorded in the “Partial freezing the UA string” review
• This work is being funded by: Google

We'd prefer the TAG provide feedback as (please delete all but the desired option):

🐛 open issues in our GitHub repo for each point of feedback

[hober]

Hi @miketaylr!

The review in issue #467 was previously closed, but I’m requesting a new review now that Chrome is interested in pursuing this plan again.

Have there been any substantive changes since our last review? If so, could you give us a summary of them, so we know what to concentrate on? If not, I'm not sure what there is to do here.

[miketaylr]

Heya @hober!

Yeah, sorry. It probably would have been useful for me to do so as I filed this 🙈 . The most substantial changes can be summed up as:

• A more fleshed out spec exists for User-Agent Client Hints
• The original "UA Freezing" ideas proposed that all non-mobile browsers should report as Windows 10 devices. We've since changed our thinking on this, for reasons articulated in the blink-dev intent to prototype & ship thread. Instead, our current thinking is to keep the platform exposed in the User-Agent string, for compatibility and accessibility reasons (sadly many sites rely on sniffing for platform to handle keyboard shortcut differences between OSes).

edit: perhaps more usefully, these are the major updates since the original intent filed by @yoavweiss, and the ones captured in the Blink Intent to Prototype & Ship:

1. Sec-CH-UA-Bitness: adds a new high-entropy hint to expose the OS bitness, which may be combined with Sec-CH-UA-Arch to provide optimized binaries for download, for example.
2. Make Sec-CH-UA-Platform a low-entropy hint: OS is passively observable at the TCP level anyways, so we plan to change this to be low-entropy and send as a default header (similar to Sec-CH-UA and Sec-CH-UA-Mobile).
3. Include low-entropy hints by default in UADataValues (returned by getHighEntropyValues()). If a hint moves from high to low-entropy, this future proofs any code relying on it.
4. Add a toJSON method to NavigatorUAData’s IDL. Technically a bugfix, but it is an API change (instead of returning {}, JSON.stringify(navigator.userAgentData)) will now be useful)

In addition, we've merged in 67 PRs to the spec since #467 was first filed (some more substantial than others).

[torgo]

An early question - where does UA Client Hints go after WICG? Is the intention to bring this into the Web Apps working group, for example? It's not clear to me what the trajectory for this is?

[marcoscaceres]

I think they are more of a Web Perf WG thing - hi @yoavweiss!

[marcoscaceres]

Having said that, we can take them in WebApps WG, but we would need to add it as a thing to our charter. Our charter is up for renewal, so it's a good time to add. However, we'd like to make sure there is a support from a second implementer and no objection from a third.

[yoavweiss]

The mental model I had was that each feature that relies on CH could go to the most relevant WG, while the infrastructure could go to WebPerfWG, or ideally integrated directly into Fetch and HTML.
For UA-CH specifically, WebApps seems appropriate.

In terms of support or lack thereof, I'll note that Mozilla considered it non-harmful and @othermaciej previously said he's personally "mildly positive" on the direction with a few reservations, but since had at least one objection. I'm hoping these reservations and objections are things that can be worked out in the WG, rather than ones that would block adoption.

[torgo]

As noted in today's call we'll try to prioritize getting back to you with some feedback on the platform issue.

[marcoscaceres]

Sent w3c/webappswg#54 to WebAppsWG to add to charter.

[torgo]

@miketaylr can you give us a bit more info on how other browsers are being engaged on this issue? I see some info about other engines but is there there active engagement?

[miketaylr]

(pardon the delay in response, in the middle of a move)

On UA Reduction:

Mozilla has taken independent steps to begin freezing information in the User-Agent string (see [1][2]), and is positive on the concept in general (see [3]). Apple has also frozen (or capped) most information in the Safari User-Agent string (see [4][5]). We would like to document how browsers have already taken these steps in the WHATWG compat standard (to start with, anyways - it may make sense for this to ultimately live in another spec). We met with Mozilla a few months back to discuss the issue of freezing the macOS version number (we invited Apple, but nobody was able to attend due to short notice). See https://www.otsukare.info/2021/03/02/capping-user-agent-string for minutes.

Personally, I would welcome participation from any browser vendor or interested party to help here: https://github.com/whatwg/compat/issues?q=is%3Aissue+is%3Aopen+%5Buastring%5D

On UA-CH:

Mozilla is neither positive nor negative on User-Agent Client Hints ("non-harmful"). We don't have a formal position from Apple one way or the other, but @othermaciej has filed a lot of high-quality bugs on the spec (only 1 out of 16 remain unresolved, see [6]). Edge has stated that they are supportive of User-Agent Client Hints (see [7]), and have also filed some high-quality bugs on the spec (two of which I hope to address soon, see [8]).

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1679929
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1693295
[3] mozilla/standards-positions#202 (comment)
[4] https://webkit.org/blog/8042/release-notes-for-safari-technology-preview-46/
[5] https://twitter.com/__jakub_g/status/1398324307814752256
[6] https://github.com/WICG/ua-client-hints/issues?q=is%3Aissue+author%3Aothermaciej
[7] https://twitter.com/_scottlow/status/1206831008261132289
[8] https://github.com/WICG/ua-client-hints/issues?q=is%3Aissue+author%3Aerik-anderson+

[torgo]

We've taken note of Mozilla's recent feedback. It looks like information is coming from implementers which would call for more thorough review. @miketaylr do you have any response to Mozilla's position?

[torgo]

Hi @miketaylr can you give us an update on this? It looks like Mozilla's position has shifted and https://mozilla.github.io/standards-positions/#ua-client-hints it looks like you're shipping this in Chrome 98 https://chromestatus.com/feature/5703317813460992. Have the issues that were raised in the TAG session been taken into account? If so, we'll likely close this review positively.

[miketaylr]

Hi @torgo, thanks for the ping. Yes, I believe we've taken Mozilla's feedback into account, both by shipping the Sec-CH-UA-Full-Version-List hint, as well as trying to make the spec more clear that it does not require browsers to expose more entropy than they currently do in the User-Agent string.

[jorabin-51d]

We think that what is proposed is a very major change to the way the Web works. On the face of it what is contemplated is "merely" a change to the use of the User-Agent HTTP header field and introduction of other HTTP fields, altering the interaction between client and server.

However the consequences are much wider than this and affect almost all the related systems that go to make the Web work. We are therefore very concerned that the proposals will “break the Web” in various ways, and therefore require very close scrutiny.

• Use of the User-Agent field in its present form is understood to be an accretion of various custom and practice over many years, it is untidy but it works.
• It’s understood to be a potential privacy problem as potentially exploited by unscrupulous Web site operators. It’s said to be a cause of mis-operation. It’s said to be the cause of other problems, however evidence has not been presented as to the extent or severity of any of such effects.
• It’s known to be useful to honest Web site operators in detecting fraud and other unscrupulous behaviour by fraudulent Web client software, robots etc. These anti-fraud applications will cease to operate in their present form.
• It is known to be widely used by benign applications to support the effective operation of the Web in other ways - e.g. CDN has been mentioned in this context. These applications also require the User-Agent in its present form.

We think that proper standardisation scrutiny of changes to established standards and their established use is required:

• The changes proposed in the form in which they have been proposed have not been justified – for example browser vendors are free to change their User-Agent header to reduce passive profiling, no change to HTTP headers is needed.
• We have not seen a measured argument of how the supposed positive effects of any such changes are to be weighed against the undoubtedly negative effects of the changes. The form in which changes have been proposed stand a good chance of “breaking the Web”.
• None of the proposals entailed, either at IETF or at W3C have been proposed for standards track scrutiny. Because of the highly disruptive potential of these changes, standardisation scrutiny is required.

We note that the changes discussed, although still in draft and unstandardised form are finding their way into live implementations.

We will be grateful for TAG consideration of these points.

Many thanks

Jo Rabin
CTO 51Degrees

[jorabin-51d]

We read that Google is proceeding with this programme as of release M101 which we understand to be scheduled for the start of Q2 2022. We are very concerned about the timeframe for this introduction and its implications on our customers and their users and the Web in general. As we describe below, we consider the timeframe to be reckless.

We operate a “device detection” service which our customers use for various purposes including enhancement of their users’ experience, fraud detection, analytics and other similar benign activities. Our customers tell us that they have not heard about or know little about the proposals. While we have adapted our product to work according to what we understand the proposals to be, our customers require time to implement any changes to their Web infrastructure and the practicality of doing so soon is very limited.

Our understanding of Google’s proposals is in any case “at a point in time”, the stability of those proposals has not yet been established (the latest version of User-Agent Client Hints is dated Feb 9) and they have not yet achieved consensus as to their nature in any standardisation forum. We were pleased to hear about the recently announced (Jan 13) User-Agent Reduction Deprecation Origin Trial that would allow maintenance of the current UA for some time, however we don’t think that our customers will have time to introduce this before Q2 given current engineering schedules and processes.

To summarise, the market is not ready for these changes. Significant parts of the Web - for example OpenRTB - which use the User-Agent header value have not been changed. The changes have the potential to cause wholesale disruption and will at least cause a considerable level of dysfunction in the established Web environment. We therefore ask Google to postpone the changes they propose, while the basis of those proposals stabilises, some form of standardisation consensus about them emerges and giving the market reasonable time to prepare for them.

We will be grateful if the TAG will please note our further views in its considerations.

Many thanks
Jo Rabin
CTO 51Degrees

[miketaylr]

We read that Google is proceeding with this programme as of release M101 which we understand to be scheduled for the start of Q2 2022.

That's correct. I posted a link in this thread last September pointing to the timelines associated with User-Agent Reduction.

[torgo]

Hi @jorabin-51d - Regarding some of the issues you raised I believe Mike and Yoav believe that these have been addressed - see the minutes of our call on 14-Jun from last year. @yoavweiss can you provide any further detail or comment here?

[jorabin-51d]

Hi @torgo
 
No, I don’t think the issues have been adequately addressed – here are some material points:
 

• We are weeks away from the unilateral implementation of a significant change to the operation of the Web
• That change hasn’t been standardised and hasn’t yet had a TAG design review
• The details of the change are still in flux (Feb 9 being the latest update)
• We see no evidence that websites are ready for the change, indeed we think that their owners are mostly unaware that a change is needed
   
  Many thanks
  Jo

[miketaylr]

@yoavweiss can you provide any further detail or comment here?

@yoavweiss is on PTO this week, but I’m happy to reply.

@jorabin-51d writes:

We are therefore very concerned that the proposals will “break the Web” in various ways, and therefore require very close scrutiny…
…The form in which changes have been proposed stand a good chance of “breaking the Web”.

I really appreciate the concern for compatibility - it was a guiding principle when we designed UA Reduction and its phased rollout. Most of the use cases mentioned above rely on hints that are available by default (user experience improvements, analytics, etc.) and will not be reduced in the User-Agent header. A site won’t need to migrate to UA-CH if they only rely on browser name, major version, platform, or mobileness. Use cases like fraud detection would require extra hints, and migrating to UA-CH will allow those to continue to function.

That said, we haven’t received concrete evidence of site compat issues through our own testing or from users who have enabled the chrome://flags/#reduce-user-agent flag. We’ve also been running an Origin Trial since Chrome 95 (launched on October 19th of last year) but have not received reports of site breakage, either publicly or through partner channels. If @jorabin-51d does know of sites that will break, we welcome reports at https://crbug.com/new so we can work with the sites to get them fixed.

@jorabin-51d also writes:

That change hasn’t been standardised…

We hope to bring UA-CH to the W3C (and have already stated as much in this thread) once another implementer has stated an interest in implementing it.

Client Hints (which UA-CH builds upon) is standardized in the IETF.

We recently started the work to document User-Agent Reduction patterns by Firefox, Chrome, and Safari in the WHATWG. There’s more work to be done there, and it’s on my plate over the coming months.

…and hasn’t yet had a TAG design review

I’ll also note that we’re having this discussion in the middle of a TAG design review. :)

[jorabin-51d]

Thanks for your reply @miketaylr

I believe that this topic still requires a lot of discussion. We stand by the points we have already made about the changes proposed being reckless and that they will cause harm to the operation of the Web and associated ecosystem of services, applications etc. Rather than repeat those points, which like I say bear further discussion, I will try to keep my response as brief as I can:

A site won’t need to migrate to UA-CH if they only rely on browser name, major version, platform, or mobileness

In general that’s not the case. One of the strongest use cases is for device type, for example.

but have not received reports of site breakage, either publicly or through partner channels. If @jorabin-51d does know of sites that will break, we welcome reports at https://crbug.com/new so we can work with the sites to get them fixed.

We don’t think that gross mis-operation in the form of 500 errors etc. is the primary concern. Of more concern is user experience and optimisation. Site owners are always keen to respond quickly on first request with the best experience possible. I’m not sure I see the point of millions of sites reporting that the initial and subsequent experience for their users has become poorer, since that is a feature of the design.

We hope to bring UA-CH to the W3C (and have already stated as much in this thread) once another implementer has stated an interest in implementing it.

It is a very significant concern that there is no other implementer, since this means that the Chromium way of doing things will splinter the Web. There is a good chance that this fracture in the Web will continue indefinitely. Were another implementer to come on board it seems likely to me that the standard would move on from today’s specification with potential further disruption to website operators. We face a world of “the chromium way” and “the standard way”. We know the TAG takes a strong interest in proposals being broadly supported and of wide applicability, and we think that the evidence says this is a standard that is not reached by this proposal.

Client Hints (which UA-CH builds upon) is standardized in the IETF.

Noting that this RFC is “Experimental” and is therefore not standardised at IETF. For the convenience, here is what IETF says (RFC 2026) about Experimental:

Specifications that are not on the standards track are labeled with one of three "off-track" maturity levels: "Experimental", "Informational", or "Historic". The documents bearing these labels are not Internet Standards in any sense.

I’ll also note that we’re having this discussion in the middle of a TAG design review. :)

Noted. However the plan to proceed appears to be independent of what this TAG review will conclude.

Once again hoping that the TAG will find the content of this dialogue informative as to its findings.

Many thanks

[miketaylr]

Hi @jorabin-51d,

A site won’t need to migrate to UA-CH if they only rely on browser name, major version, platform, or mobileness

In general that’s not the case. One of the strongest use cases is for device type, for example.

Can you clarify what you mean by device type?

We don’t think that gross mis-operation in the form of 500 errors etc. is the primary concern. Of more concern is user experience and optimisation.

I agree that 500s are not the only kinds of concerning site breakage (and in my experience 400-level errors are more common than 500-level for UA changes). If you know about concrete UX or optimization bugs, or just things generally not working as expected please file them.

It is a very significant concern that there is no other implementer, since this means that the Chromium way of doing things will splinter the Web. There is a good chance that this fracture in the Web will continue indefinitely.

It's fairly normal for one engine to ship ahead of other engines, depending on the feature. That's not a new thing. Chrome is actually lagging when it comes to User-Agent Reduction efforts: Safari gets a gold medal from me 🥇 for being the leader, I think, with Firefox coming in second place 🥈 (reasonable people can disagree with the ordering there 🤷 ).

But I will note that other browsers like Firefox and Safari have not offered a way to recover the information lost to reduction. I'm not sure that's a better outcome for sites. That said, I'm hopeful they will see the value in the UA-CH API and eventually implement it, and am happy to work with them on the spec and in whatever eventual working group it ends up in to address their feedback.

Were another implementer to come on board it seems likely to me that the standard would move on from today’s specification with potential further disruption to website operators.

Are you saying another implementer is going to make things worse? Or am I missing the point?

Noting that this RFC is “Experimental” and is therefore not standardised at IETF.

OK. I think we have different definitions (and spellings!) in mind for standardization.

However the plan to proceed appears to be independent of what this TAG review will conclude.

Maybe there's some misunderstanding of what role TAG a review plays in the Blink Process. The Chromium project asks for the TAG's review and advice when it comes to technical designs and how they fit into the broader platform, not for advice on product decisions like shipping or timelines.

(An early review by the TAG seemed pretty positive on the concept in general, FWIW.)

[jorabin-51d]

Hi @miketaylr

I believe I have made my points and while I am always open to discussion regret that as a small company I do not have the resources to continue this to and fro in this forum.

[torgo]

Hi -

We are going to close this review with an outcome of "satisfied with concerns." In the main, we are happy with this proposal including the proposed use of client hints to make up for the lack if discoverability via the UA string.

The concerns we still have are about multi-stakeholder, compatibility and standardisation venue.

We are concerned about multi-implementation. We would like to see this taken up by other user agents.

We would like to the proposers to ensure that they are receptive to the concerns raised in the community regarding web content compatibility. We urge you to be transparent about publishing results of studies regarding compatibility. We also encourage the community to raise concerns in a transparent and dispassionate manner.

Although we're satisfied that the client hints spec itself is on a firm standards track footing, we're concerned about the venue for the standardisation of the ua-client-hints spec. We would like to see the ua-client-hints spec move to a proper w3c working group such as Web Applications.

[yoavweiss]

Apologies for my delayed reply here... I fully agree with what @miketaylr said.