feat(webtransport): add WebTransport for WASM environments #4015

oblique · 2023-05-31T13:18:42Z

Description

This PR implements Transport for WebTransport for browsers by using web-sys.

Related: #3846.
Resolves: #3825.

Notes & open questions

This PR depends on #3991 and on latest master branch of multiaddr

WASM in browsers is single threaded and multi-threading is still work in progress with a lot of unknowns. Currently wasm_bindgen_futures crate implements only spawn_local, which is what is used by libp2p_swarm::executor::WasmBindgenExecutor. However because I needed to satisfy Send, I used send_wrapper crate.

If in the future multithreading is introduced in browser's WASM, then this code might need refactoring.

I was thinking to also move the current code of web-ext/src in web-ext/src/websockets, but I didn't do it.

Documentation will be added soon, but you can review the code.

Change checklist

I have performed a self-review of my own code
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
A changelog entry has been made in the appropriate crates

oblique · 2023-05-31T13:20:03Z

This includes the commits of #3991 and I will rebase after it gets merged

mxinden

Thank you @oblique for the high quality contribution. I am looking forward to landing this.

Overall this looks good to me. I still need some more time for another review.

transports/wasm-ext/src/webtransport/transport.rs

mxinden · 2023-06-01T05:44:43Z

transports/wasm-ext/src/webtransport/transport.rs

+impl Transport {
+    pub fn boxed(self) -> Boxed<(PeerId, StreamMuxerBox)> {
+        self.map(|(peer_id, muxer), _| (peer_id, StreamMuxerBox::new(muxer)))
+            .boxed()
+    }
+}


Linking related discussion. No actions required from my end.

#3313

transports/wasm-ext/src/webtransport/substream.rs

transports/wasm-ext/src/webtransport/transport.rs

thomaseizinger

I can only 2nd @mxinden ! Very high quality, thanks a lot for this! Super excited to land this.

Left a few minor comments :)

I am also curious about your usecase and perhaps you want to add yourself to https://github.com/libp2p/rust-libp2p#notable-users.

transports/wasm-ext/src/lib.rs

transports/wasm-ext/src/webtransport.rs

transports/wasm-ext/src/lib.rs

transports/wasm-ext/src/webtransport/cached_js_promise.rs

transports/wasm-ext/src/webtransport/error.rs

transports/wasm-ext/src/webtransport/transport.rs

transports/wasm-ext/src/webtransport/substream.rs

transports/wasm-ext/src/webtransport/connection.rs

transports/websys-webtransport/src/connection.rs

Cargo.toml

transports/webtransport-websys/Cargo.toml

mxinden · 2023-06-05T08:53:28Z

transports/webtransport-websys/src/connection.rs

+        // TODO: This should be part libp2p-noise
+        if let Some(expected_peer_id) = remote_peer {


Fine by me. Though I suggest doing so in a separate pull request. Also note that we do not always know the remote's peer id in advance.

Related: #2946

To actually make use of this in our upgrade stack, some more changes are required.

@thomaseizinger I was actually thinking to do the following:

Add libp2p_noise::Config::with_remote_peer_id

The above will pass the value at libp2p_noise::io::handshake::State::expected_remote_peer_id

In libp2p_noise::io::handshake::State::finish we will verify if id_pk.to_peer_id() is the same with State::expected_remote_peer_id

After this change I would be able to do some like this in the Connection::authenticate:

let mut noise = libp2p_noise::Config::new(keypair)?; if !certhashes.is_empty() { noise = noise.with_webtransport_certhashes(certhashes); } if let Some(expected_peer_id) = remote_peer { noise = noise.with_remote_peer_id(expected_peer_id); }

What do you think? I can do it in separate PR

Yes sounds good to me!

I only linked #2946 here because it would be great if we can take advantage of this built-in verification for all connection upgrades and not just here.

What do you think? I can do it in separate PR

Preference for separate pull request.

.cargo/config.toml

transports/webtransport-websys/src/stream.rs

transports/webtransport-websys/src/utils.rs

.github/workflows/ci.yml

transports/webtransport-websys/src/lib.rs

thomaseizinger · 2023-06-05T10:12:47Z

transports/webtransport-websys/src/connection.rs

+        // TODO: This should be part libp2p-noise
+        if let Some(expected_peer_id) = remote_peer {


Related: #2946

To actually make use of this in our upgrade stack, some more changes are required.

mxinden · 2023-06-06T10:45:30Z

@oblique on Slack you were asking how to best test this pull request. @jxs suggested the following:

We have an interoperability test framework at https://github.com/libp2p/test-plans/tree/master/multidim-interop
We test interoperability between js-libp2p running in the browser connecting to a go-libp2p server via WebTransport
- We use a headless browser
- You can find the implementation here https://github.com/libp2p/test-plans/tree/master/multidim-interop/impl/js/v0.44
- It is referenced and thus picked up by the framework here https://github.com/libp2p/test-plans/blob/master/multidim-interop/versions.ts#L94-L100 (note the { name: "webtransport", onlyDial: true })
You could add a similar implementation, just using rust-libp2p compiled to WASM, via a headless browser to the rust-libp2p interop implementation https://github.com/libp2p/rust-libp2p/tree/master/interop-tests
And reference it in https://github.com/libp2p/test-plans/blob/master/multidim-interop/versions.ts

This would test interoperability between a rust-libp2p WASM WebTransport node (this pull request) against any other libp2p based WebTransport server implementation on every pull request.

I know this looks rather complex at first. Please either comment here, ping me on Slack, Matrix or e-mail, or let's schedule a quick call sometime.

What do you think?

oblique · 2023-06-06T17:56:59Z

@mxinden This looks complex indeed and I'm not sure how much time I can devote for it right now. I am not familiar with JavaScript ecosystem which is something that is needed for the Rust WASM case.

I will spend some time to figure some stuff out, but I can not guarantee that I will be able to deliver the interop tests.

In case of the Go echo-server, I was thinking to do much simpler tests and closer to the Transport layer than the Swarm layer.

Approvals have been dismissed because the PR was updated after the send-it label was applied.

oblique · 2023-06-23T06:09:07Z

Thanks a lot!! 🎉

zvolin · 2023-06-23T09:26:18Z

Btw, @thomaseizinger it looks like my contribution isn't counted by github 😢 is it because it's Co-Authored-By instead of Co-authored-by?

mxinden · 2023-06-23T09:36:02Z

@thomaseizinger @oblique @zvolin in case it is of some use, I can release libp2p v0.52.1 and libp2p-webtransport-websys v0.1.0.

oblique · 2023-06-23T09:43:42Z

@thomaseizinger @oblique @zvolin in case it is of some use, I can release libp2p v0.52.1 and libp2p-webtransport-websys v0.1.0.

yes please

zvolin · 2023-06-23T09:45:15Z

I guess there are no chances you amend the message so that my contribution is properly attributed? 😄

zvolin · 2023-06-23T09:51:30Z

#4104 Here is the fix for a future contribs

thomaseizinger · 2023-06-23T10:49:41Z

Btw, @thomaseizinger it looks like my contribution isn't counted by github 😢 is it because it's Co-Authored-By instead of Co-authored-by?

I am terribly sorry for this. I am pretty sure, GitHub attributed it correctly for b8ceecc but now that one is also missing.

Maybe GitHub is buggy here? It is kind of insane that the casing matters ..
I'll open an issue.

thomaseizinger · 2023-06-23T10:53:02Z

I guess there are no chances you amend the message so that my contribution is properly attributed? 😄

I'd rather not but maybe we can get GitHub to fix this and then it should be counted retroactively.

zvolin · 2023-06-23T11:17:44Z

I'd rather not but maybe we can get GitHub to fix this and then it should be counted retroactively.

chances are rather poor, but thanks, would appreciate updates how it went out

thomaseizinger · 2023-06-23T18:25:53Z

I'd rather not but maybe we can get GitHub to fix this and then it should be counted retroactively.

chances are rather poor, but thanks, would appreciate updates how it went out

Just got a reply, apparently the issue are the blank lines but that does not explain why b8ceecc is not correctly attributed. Still waiting for a response to that.

mxinden · 2023-06-26T00:43:10Z

@thomaseizinger @oblique @zvolin in case it is of some use, I can release libp2p v0.52.1 and libp2p-webtransport-websys v0.1.0.

yes please

@oblique @zvolin published ✔️

…rt test Introduced in rust-libp2p with libp2p/rust-libp2p#4015.

zvolin · 2023-07-03T13:33:19Z

Just got a reply, apparently the issue are the blank lines but that does not explain why b8ceecc is not correctly attributed. Still waiting for a response to that.

btw, any new info from github regarding this @thomaseizinger ? I still have some hope 😄

thomaseizinger · 2023-07-03T13:55:17Z

Just got a reply, apparently the issue are the blank lines but that does not explain why b8ceecc is not correctly attributed. Still waiting for a response to that.

btw, any new info from github regarding this @thomaseizinger ? I still have some hope 😄

No new information. They referring me to the git interpret-trailers command but running the commit message through that one correctly parses the trailer so I am not sure why the contribution did not work for that one.

Regarding the original commit, they only referred to documentation on how to edit a commit message, which requires force-pushing. Not really a solution in my opinion. Force-pushing master is overall considered an anti-pattern so it is sad that they don't allow for an additional way of specifying co-authorship. After looking into this a bit more, it seems that interpreting trailers is extremely whitespace sensitive which is quite annoying when trying to generate the commit message automatically.

thomaseizinger · 2023-07-03T14:05:07Z

I've captured all the requirements in #4152.

Introduced in rust-libp2p with libp2p/rust-libp2p#4015.

When you enable Kademlia in a large network that uses multiple type of transports, then this log creates a very noisy console and makes meaningful logs hard to see. We can also remove the log entirely if you prefer. Check: #4015 (comment), #4072, #4133 Pull-Request: #5396.

When you enable Kademlia in a large network that uses multiple type of transports, then this log creates a very noisy console and makes meaningful logs hard to see. We can also remove the log entirely if you prefer. Check: libp2p#4015 (comment), libp2p#4072, libp2p#4133 Pull-Request: libp2p#5396.

oblique force-pushed the webtransport-wasm-ext branch from 494f482 to 6b854d6 Compare May 31, 2023 20:52

mxinden mentioned this pull request Jun 1, 2023

feat(noise): add WebTransport certhashes extension #3991

Merged

4 tasks

mxinden reviewed Jun 1, 2023

View reviewed changes

thomaseizinger reviewed Jun 1, 2023

View reviewed changes

oblique force-pushed the webtransport-wasm-ext branch 3 times, most recently from 928731b to a00f118 Compare June 1, 2023 15:16

oblique changed the title ~~feat(wasm-ext): Add webtransport::Transport~~ feat(transport): Add WebTransport for WASM environments Jun 1, 2023

oblique force-pushed the webtransport-wasm-ext branch 5 times, most recently from 5346e01 to ff3761d Compare June 2, 2023 14:53

oblique commented Jun 2, 2023

View reviewed changes

transports/websys-webtransport/src/connection.rs Outdated Show resolved Hide resolved

oblique force-pushed the webtransport-wasm-ext branch 4 times, most recently from 1932e08 to 231c032 Compare June 3, 2023 18:35

oblique commented Jun 3, 2023

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

oblique force-pushed the webtransport-wasm-ext branch 3 times, most recently from 3f31979 to c9c9f6c Compare June 4, 2023 20:48

mxinden reviewed Jun 5, 2023

View reviewed changes

thomaseizinger reviewed Jun 5, 2023

View reviewed changes

oblique force-pushed the webtransport-wasm-ext branch 3 times, most recently from 70c2268 to 980a12b Compare June 6, 2023 08:25

mxinden approved these changes Jun 23, 2023

View reviewed changes

mergify bot merged commit 85a846a into libp2p:master Jun 23, 2023

mxinden mentioned this pull request Jun 23, 2023

chore: prepare libp2p v0.52.1 #4106

Merged

4 tasks

mxinden mentioned this pull request Jun 24, 2023

transport/webtransport: Add WebTransport protocol #2993

Open

thomaseizinger mentioned this pull request Jun 26, 2023

webrtc: WASM support #4111

Closed

mxinden added a commit to libp2p/test-plans that referenced this pull request Jul 3, 2023

feat(interop/rust-chromium): add rust-libp2p chromium WASM WebTranspo…

505a552

…rt test Introduced in rust-libp2p with libp2p/rust-libp2p#4015.

mxinden added a commit to libp2p/test-plans that referenced this pull request Jul 3, 2023

feat(interop/rust-chromium): add rust-libp2p chromium WASM WebTranspo…

1dfa9aa

…rt test Introduced in rust-libp2p with libp2p/rust-libp2p#4015.

mxinden mentioned this pull request Jul 3, 2023

feat(interop): add rust-libp2p chromium WASM WebTransport libp2p/test-plans#217

Merged

mxinden added a commit to libp2p/test-plans that referenced this pull request Jul 7, 2023

feat(interop): add rust-libp2p chromium WASM WebTransport (#217)

ce83afc

Introduced in rust-libp2p with libp2p/rust-libp2p#4015.

mxinden mentioned this pull request Jul 10, 2023

Technical deep-dive: Retrofitting "backpressure" in yamux libp2p/blog#90

Closed

mxinden mentioned this pull request Jul 27, 2023

AutoNATv2 implementation for rust-libp2p filecoin-project/devgrants#1421

Closed

thomaseizinger mentioned this pull request Aug 19, 2023

WASM support #2617

Open

oblique mentioned this pull request May 17, 2024

chore(webtransport-websys): Change logs to debug level #5396

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(webtransport): add WebTransport for WASM environments #4015

feat(webtransport): add WebTransport for WASM environments #4015

oblique commented May 31, 2023 •

edited by thomaseizinger

Loading

oblique commented May 31, 2023 •

edited

Loading

mxinden left a comment

mxinden Jun 1, 2023

thomaseizinger left a comment

mxinden Jun 5, 2023

thomaseizinger Jun 5, 2023

oblique Jun 5, 2023

thomaseizinger Jun 5, 2023

mxinden Jun 9, 2023

thomaseizinger Jun 5, 2023

mxinden commented Jun 6, 2023 •

edited

Loading

oblique commented Jun 6, 2023

oblique commented Jun 23, 2023

zvolin commented Jun 23, 2023

mxinden commented Jun 23, 2023

oblique commented Jun 23, 2023

zvolin commented Jun 23, 2023

zvolin commented Jun 23, 2023 •

edited

Loading

thomaseizinger commented Jun 23, 2023

thomaseizinger commented Jun 23, 2023

zvolin commented Jun 23, 2023

thomaseizinger commented Jun 23, 2023

mxinden commented Jun 26, 2023

zvolin commented Jul 3, 2023

thomaseizinger commented Jul 3, 2023

thomaseizinger commented Jul 3, 2023

		// TODO: This should be part libp2p-noise
		if let Some(expected_peer_id) = remote_peer {

feat(webtransport): add WebTransport for WASM environments #4015

feat(webtransport): add WebTransport for WASM environments #4015

Conversation

oblique commented May 31, 2023 • edited by thomaseizinger Loading

Description

Notes & open questions

Change checklist

oblique commented May 31, 2023 • edited Loading

mxinden left a comment

Choose a reason for hiding this comment

mxinden Jun 1, 2023

Choose a reason for hiding this comment

thomaseizinger left a comment

Choose a reason for hiding this comment

mxinden Jun 5, 2023

Choose a reason for hiding this comment

thomaseizinger Jun 5, 2023

Choose a reason for hiding this comment

oblique Jun 5, 2023

Choose a reason for hiding this comment

thomaseizinger Jun 5, 2023

Choose a reason for hiding this comment

mxinden Jun 9, 2023

Choose a reason for hiding this comment

thomaseizinger Jun 5, 2023

Choose a reason for hiding this comment

mxinden commented Jun 6, 2023 • edited Loading

oblique commented Jun 6, 2023

oblique commented Jun 23, 2023

zvolin commented Jun 23, 2023

mxinden commented Jun 23, 2023

oblique commented Jun 23, 2023

zvolin commented Jun 23, 2023

zvolin commented Jun 23, 2023 • edited Loading

thomaseizinger commented Jun 23, 2023

thomaseizinger commented Jun 23, 2023

zvolin commented Jun 23, 2023

thomaseizinger commented Jun 23, 2023

mxinden commented Jun 26, 2023

zvolin commented Jul 3, 2023

thomaseizinger commented Jul 3, 2023

thomaseizinger commented Jul 3, 2023

oblique commented May 31, 2023 •

edited by thomaseizinger

Loading

oblique commented May 31, 2023 •

edited

Loading

mxinden commented Jun 6, 2023 •

edited

Loading

zvolin commented Jun 23, 2023 •

edited

Loading