proposal: Noise extension registry

[marten-seemann]

Problem: Noise doesn’t define a way to add general-purpose extensions to the handshake. These extensions could be used to negotiate optional features.

Currently, we have 3 (!) proposals in flight that would use Noise’s Early Data feature to either modify the Noise handshake or negotiate application-layer behavior:

1. WebTransport: webtransport: use CBOR to encode cert hashes during Noise handshake #429, to tie the certificate hash to the (authenticated) peer IDs
2. WebRTC: webrtc/: Add libp2p WebRTC browser-to-server spec #412, in a similar fashion, to tie the certificate fingerprint to the peer IDs
3. Muxer selection in security handshake #446, to negotiate the stream multiplexer

By pure coincidence, these 3 use cases don’t overlap, i.e. you’d only want to use one OR the other, but not two or three at the same time. However, I can easily come up with use cases where you’d want to use one of those described above AND another extension.

As a point of reference, both TLS (via RFC 6066) and QUIC (via transport parameters, see Section 19.21 of RFC 9000 provide such a general-purpose mechanism.

I propose to define a mechanism exactly along those lines: When defining a Noise extension, you’d pick a code point, and could then attach extension-defined data to that code point. It makes sense to use a Protobuf here, as our Noise handshake already uses Protobufs. We’re making use of the property of Protobufs that only fields that actually have a value are being serialized (I assume).

With the 3 use cases before, we’d start with something like this:

message NoiseExtension {
    repeated bytes webtransport_certhashes = 1;
    optional bytes webrtc_fingerprint = 2;
    repeated string stream_muxers = 3; 
}

We could also add version field at the beginning of the protobuf, with a fixed version number, to make this even more future-proof.

Regarding document structure, this definition would probably live in a separate markdown file in the Noise directory. Specifications that want to use a Noise extension would need to update this file.

Regarding code points, we can reserve the short (1- and 2-byte varints) for approved specifications, and leave the space with longer varints open for experimentation on a first-come-first-serve basis.

@MarcoPolo @mxinden @julian88110 @p-shahi @BigLep, what do you think?

[Menduist]

I already mentioned this somewhere, but I think we could push this even further, and create a generic "framework" to pass early data to upper layers
It could be used in noise, protocol select, every "wrapper" layer really

Then, in noise & co, we just have to define where to put the data
And in upper layers, we have to define the id of the data, and it's format

One thing to keep in mind is forward-compatibility. How will this data evolve for future versions of muxers, webrtc & webtransport, how we can communicate which version we are using, and did we manage to consume the data sent by the peer, or are we not compatible

[MarcoPolo]

We’re making use of the property of Protobufs that only fields that actually have a value are being serialized (I assume).

This is true

This seems like a good idea @marten-seemann, thanks for writing this up. The one addition I would add to this is that we version this by prefixing the binary data with a varint instead of a version field within the protobuf. The reason for this is so that when we parse the protobuf we know what version of the message to expect instead of having to do two passes of the encoded data. This version varint prefix can be a multicodec, but I'm not sure there's much to gain (we only ever do this in noise) and it could result in a 2 byte version prefix rather than a single byte version prefix (depending on which codec we register).

I already mentioned this somewhere, but I think we could push this even further, and create a generic "framework" to pass early data to upper layers
It could be used in noise, protocol select, every "wrapper" layer really

@Menduist can you give me an example of when another layer would do this? This seems pretty specific to the noise handshake (In TLS we're using the ALPN to early negotiate the muxer, and that's a different format). I'm not sure what it means for a muxer to get/send early data from the remote and propogate it up to the stream abstraction, but maybe I'm missing something.

[marten-seemann]

@Menduist

One thing to keep in mind is forward-compatibility. How will this data evolve for future versions of muxers, webrtc & webtransport, how we can communicate which version we are using

If we need to encode data differently for WebTransport (say) in the future, we can reserve a new code point for that.

and did we manage to consume the data sent by the peer, or are we not compatible

That's for the extension to define. Many TLS extensions have the server reply something (sometimes even an empty extension) to signal support for a particular extension that the client offered. But we don't need to define that here.

@MarcoPolo

The one addition I would add to this is that we version this by prefixing the binary data with a varint instead of a version field within the protobuf. The reason for this is so that when we parse the protobuf we know what version of the message to expect instead of having to do two passes of the encoded data. This version varint prefix can be a multicodec, but I'm not sure there's much to gain (we only ever do this in noise) and it could result in a 2 byte version prefix rather than a single byte version prefix (depending on which codec we register).

That makes sense. This would also allow us to switch to a different encoding than Protobuf in the future, if we ever desire to do so.

[julian88110]

Thanks for floating this. This gives a coordinated solution between different efforts. One thing is that currently the early data in Noise is carried as opaque byte data, https://github.com/libp2p/specs/tree/master/noise#the-libp2p-handshake-payload, I assume there is no backward compatibility issue if we change the format?

[marten-seemann]

One thing is that currently the early data in Noise is carried as opaque byte data, https://github.com/libp2p/specs/tree/master/noise#the-libp2p-handshake-payload, I assume there is no backward compatibility issue if we change the format?

We haven't been using early data so far, so this is the moment to define how the opaque data will look like :)

[marten-seemann]

I just realized that we can just re-define data field in the NoiseHandshakePayload protobuf (or maybe just bump the ID from 3 to 4?). That way, this could all be a single (composed) protobuf, and we wouldn’t need to worry about versioning and putting multicodecs in there.
I’ll create a PR to the spec.

[mxinden]

Thanks for pushing this forward @marten-seemann.

Currently, we have 3 (!) proposals

🎉

Currently, we have 3 (!) proposals in flight that would use Noise’s Early Data feature to either modify the Noise handshake or negotiate application-layer behavior:

For the sake of completeness, #412 does not leverage Noise's early data feature, but Noise's prologue feature. In other words, this additional data is never exchanged on the wire in any of the 3 handshake messages, but instead just required to be the same on each side.

[marten-seemann]

One obvious use case for the extension registry, providing an easy way speed up the connection establishment, would be to put the list of supported protocols of Identify into a Noise extension.

For the sake of completeness, #412 does not leverage Noise's early data feature, but Noise's prologue feature.

Thank you @mxinden! I removed the WebRTC entry from #453. WebRTC's use of Prologue should probably be documented in the Noise spec, but this is completely orthogonal to this proposal.