Jco transpile: implementation of wasi-sockets for Node.js

[pchickey]

Tasks

• instance-network.instance-network
• ip-name-lookup.resolve-addresses
• ip-name-lookup.resolve-next-address
• ip-name-lookup.drop-resolve-address-stream
• ip-name-lookup.subscribe
• tcp-create-socket.create-tcp-socket
• tcp.start-bind
• tcp.finish-bind
• tcp.start-connect
• tcp.finish-connect
• tcp.start-listen
• tcp.finish-listen
• tcp.accept
• tcp.local-address
• tcp.remote-address
• tcp.is-listening
• tcp.address-family
• tcp.ipv6-only
• tcp.set-ipv6-only
• tcp.set-listen-backlog-size
• tcp.keep-alive-enabled
• tcp.set-keep-alive-enabled
• tcp.keep-alive-idle-time
• tcp.set-keep-alive-idle-time
• tcp.keep-alive-interval
• tcp.set-keep-alive-interval
• tcp.keep-alive-count
• tcp.set-keep-alive-count
• tcp.receive-buffer-size
• tcp.set-receive-buffer-size
• tcp.send-buffer-size
• tcp.set-send-buffer-size
• tcp.subscribe
• tcp.shutdown
• tcp[Symbol.dispose]()
• udp-create-socket.create-udp-socket
• udp.start-bind
• udp.finish-bind
• udp.start-connect
• udp.finish-connect
• udp.stream
• udp.local-address
• udp.remote-address
• udp.address-family
• udp.ipv6-only
• udp.set-ipv6-only
• udp.unicast-hop-limit
• udp.set-unicast-hop-limit
• udp.receive-buffer-size
• udp.set-receive-buffer-size
• udp.send-buffer-size
• udp.set-send-buffer-size
• udp.subscribe
• udp[Symbol.dispose]()
• incoming-datagram-stream.receive
• incoming-datagram-stream.subscribe
• outgoing-datagram-stream.send
• outgoing-datagram-stream.subscribe

Conformance Tests

IP Lookup (1/1 test)

• ℹ️ preview2_ip_name_lookup (may fail if nameserver is not set correctly - see /etc/resolv.conf)

TCP (??/46 tests)

• ✅preview2_tcp_bind
  • ✅ test_tcp_bind_ephemeral_port(&net, IpAddress::IPV4_LOOPBACK);
  • ✅ test_tcp_bind_ephemeral_port(&net, IpAddress::IPV6_LOOPBACK);
  • ✅ test_tcp_bind_ephemeral_port(&net, IpAddress::IPV4_UNSPECIFIED);
  • ✅ test_tcp_bind_ephemeral_port(&net, IpAddress::IPV6_UNSPECIFIED);
  • ✅ test_tcp_bind_specific_port(&net, IpAddress::IPV4_LOOPBACK);
  • ✅ test_tcp_bind_specific_port(&net, IpAddress::IPV6_LOOPBACK);
  • ✅ test_tcp_bind_specific_port(&net, IpAddress::IPV4_UNSPECIFIED);
  • ✅ test_tcp_bind_specific_port(&net, IpAddress::IPV6_UNSPECIFIED);
  • ✅ test_tcp_bind_addrinuse(&net, IpAddress::IPV4_LOOPBACK);
  • ✅ test_tcp_bind_addrinuse(&net, IpAddress::IPV6_LOOPBACK);
  • ✅ test_tcp_bind_addrinuse(&net, IpAddress::IPV4_UNSPECIFIED);
  • ✅ test_tcp_bind_addrinuse(&net, IpAddress::IPV6_UNSPECIFIED);
  • ✅ test_tcp_bind_addrnotavail(&net, RESERVED_IPV4_ADDRESS);
  • ✅ test_tcp_bind_addrnotavail(&net, RESERVED_IPV6_ADDRESS);
  • ✅ test_tcp_bind_wrong_family(&net, IpAddressFamily::Ipv4);
  • ✅ test_tcp_bind_wrong_family(&net, IpAddressFamily::Ipv6);
  • ✅ test_tcp_bind_non_unicast(&net);
  • ✅ test_tcp_bind_dual_stack(&net);
• ✅ preview2_tcp_connect
  • ✅ test_tcp_connect_unspec(&net, IpAddressFamily::Ipv4);
  • ✅ test_tcp_connect_unspec(&net, IpAddressFamily::Ipv6);
  • ✅ test_tcp_connect_port_0(&net, IpAddressFamily::Ipv4);
  • ✅ test_tcp_connect_port_0(&net, IpAddressFamily::Ipv6);
  • ✅ test_tcp_connect_wrong_family(&net, IpAddressFamily::Ipv4);
  • ✅ test_tcp_connect_wrong_family(&net, IpAddressFamily::Ipv6);
  • ✅ test_tcp_connect_non_unicast(&net);
  • ⚠️ test_tcp_connect_dual_stack(&net); (test passing but an error -49 is triggered - investigate))
• ❌ preview2_tcp_sample_application
  • test_tcp_sample_application(ipv4...)
  • test_tcp_sample_application(ipv6...)
• ✅ preview2_tcp_sockopts
  • ✅ test_tcp_sockopt_defaults(IpAddressFamily::Ipv4);
  • ✅ test_tcp_sockopt_defaults(IpAddressFamily::Ipv6);
  • ✅ test_tcp_sockopt_input_ranges(IpAddressFamily::Ipv4);
  • ✅ test_tcp_sockopt_input_ranges(IpAddressFamily::Ipv6);
  • ✅ test_tcp_sockopt_readback(IpAddressFamily::Ipv4);
  • ✅ test_tcp_sockopt_readback(IpAddressFamily::Ipv6);
  • ✅ test_tcp_sockopt_inheritance(&net, IpAddressFamily::Ipv4);
  • ✅ test_tcp_sockopt_inheritance(&net, IpAddressFamily::Ipv6);
  • ✅ test_tcp_sockopt_after_listen(&net, IpAddressFamily::Ipv4);
  • ⚠️ test_tcp_sockopt_after_listen(&net, IpAddressFamily::Ipv6); (test passing but an error -89 is triggered - investigate)
• ✅ preview2_tcp_states
  • ✅ test_tcp_unbound_state_invariants(IpAddressFamily::Ipv4);
  • ✅ test_tcp_unbound_state_invariants(IpAddressFamily::Ipv6);
  • ✅ test_tcp_bound_state_invariants(&net, IpAddressFamily::Ipv4);
  • ✅ test_tcp_bound_state_invariants(&net, IpAddressFamily::Ipv6);
  • ✅ test_tcp_listening_state_invariants(&net, IpAddressFamily::Ipv4);
  • ✅ test_tcp_listening_state_invariants(&net, IpAddressFamily::Ipv6);
  • ⚠️ test_tcp_connected_state_invariants(&net, IpAddressFamily::Ipv4); (test passing but an error -89 is triggered - investigate)
  • ⚠️ test_tcp_connected_state_invariants(&net, IpAddressFamily::Ipv6); (test passing but an error -89 is triggered - investigate)

UDP (??/33 tests)

• preview2_udp_bind
  • ✅ test_udp_bind_ephemeral_port(&net, IpAddress::IPV4_LOOPBACK);
  • ✅ test_udp_bind_ephemeral_port(&net, IpAddress::IPV6_LOOPBACK);
  • ✅ test_udp_bind_ephemeral_port(&net, IpAddress::IPV4_UNSPECIFIED);
  • ✅ test_udp_bind_ephemeral_port(&net, IpAddress::IPV6_UNSPECIFIED);
  • ⚠️ test_udp_bind_specific_port(&net, IpAddress::IPV4_LOOPBACK); (may fail on WSL when concurrent invocations to sock.blocking_bind())
  • ✅ test_udp_bind_specific_port(&net, IpAddress::IPV6_LOOPBACK);
  • ✅ test_udp_bind_specific_port(&net, IpAddress::IPV4_UNSPECIFIED);
  • ✅ test_udp_bind_specific_port(&net, IpAddress::IPV6_UNSPECIFIED);
  • ✅ test_udp_bind_addrinuse(&net, IpAddress::IPV4_LOOPBACK);
  • ✅ test_udp_bind_addrinuse(&net, IpAddress::IPV6_LOOPBACK);
  • ✅ test_udp_bind_addrinuse(&net, IpAddress::IPV4_UNSPECIFIED);
  • ✅ test_udp_bind_addrinuse(&net, IpAddress::IPV6_UNSPECIFIED);
  • ✅ test_udp_bind_addrnotavail(&net, RESERVED_IPV4_ADDRESS);
  • ✅ test_udp_bind_addrnotavail(&net, RESERVED_IPV6_ADDRESS);
  • ✅ test_udp_bind_wrong_family(&net, IpAddressFamily::Ipv4);
  • ✅ test_udp_bind_wrong_family(&net, IpAddressFamily::Ipv6);
  • ✅ test_udp_bind_dual_stack(&net);
• ❌ preview2_udp_connect
  • test_udp_connect_disconnect_reconnect(&net, IpAddressFamily::Ipv4);
  • test_udp_connect_disconnect_reconnect(&net, IpAddressFamily::Ipv6);
  • test_udp_connect_unspec(&net, IpAddressFamily::Ipv4);
  • test_udp_connect_unspec(&net, IpAddressFamily::Ipv6);
  • test_udp_connect_port_0(&net, IpAddressFamily::Ipv4);
  • test_udp_connect_port_0(&net, IpAddressFamily::Ipv6);
  • test_udp_connect_wrong_family(&net, IpAddressFamily::Ipv4);
  • test_udp_connect_wrong_family(&net, IpAddressFamily::Ipv6);
  • test_udp_connect_dual_stack(&net);
• ❌ preview2_udp_sample_application
  • test_udp_sample_application(ipv4...)
  • test_udp_sample_application(ipv6...)
  • test_udp_dual_stack_conversation()
• ✅ preview2_udp_sockopts
  • ✅ test_udp_sockopt_defaults(IpAddressFamily::Ipv4);
  • ✅ test_udp_sockopt_defaults(IpAddressFamily::Ipv6);
  • ✅ test_udp_sockopt_input_ranges(IpAddressFamily::Ipv4);
  • ✅ test_udp_sockopt_input_ranges(IpAddressFamily::Ipv6);
  • ✅ test_udp_sockopt_readback(IpAddressFamily::Ipv4);
  • ✅ test_udp_sockopt_readback(IpAddressFamily::Ipv6);
• ✅ preview2_udp_states
  • ✅ test_udp_unbound_state_invariants(IpAddressFamily::Ipv4);
  • ✅ test_udp_unbound_state_invariants(IpAddressFamily::Ipv6);
  • ✅ test_udp_bound_state_invariants(&net, IpAddressFamily::Ipv4);
  • ✅ test_udp_bound_state_invariants(&net, IpAddressFamily::Ipv6);
  • ✅ test_udp_connected_state_invariants(&net, IpAddressFamily::Ipv4);
  • ✅ test_udp_connected_state_invariants(&net, IpAddressFamily::Ipv6);

[manekinekko]

Note 1: Node.js does not expose libuv's setsocketopt() and getsocketopt(). We either submit a PR to Node.js or use 3rd party packages (ffi-napi and ref-napi), to expose socket libuv's bindings.

Note 2: what should be the exact behavior of client.stream(None)?

[manekinekko]

Notes from our last call on 27 Nov:

• Merge PR feat: add impl for node sockets (wip) #214 as is to keep track of current progress on the confromance tests.
• Investigate weither we could use node:net and node:dgram for the TCP and UDP impl.
• Ask more clarity about client.stream(None) (on the wasi-sockets repo).
• We need to be able to support other JS runtimes, so avoid as much as we can depending on Node.js's libuv bindings.

@pchickey @guybedford @yoshuawuyts let's keep this issue open to track futures PRs.

[manekinekko]

Following our call from last week, here is a comparison of the Node.js socket APIs from node:net/node:dgram and the Node.js libuv's binding tcp_wrap/udp_wrap. I did my best to match the Node.js APIs to the libuv APIs, but I may have missed some. Please let me know if you think so.

As you can see from the table below, there are some APIs that are not exposed in node:net, for TCP: bind() and accept(). For UDP, both node:dgram and udp_wrap expose the same APIs.

After giving this some thought, and in order to meet our deadline, I think we should keep the current wasi-sockets implementation, based on tcp_wrap as is for now. We can still add support for other runtimes in the future, once we have a stable implementation for Node.js.

TCP

API	Node.js node:net (TCP)	Node.js binding tcp_wrap	Notes
*-bind	N/A	TCP.bind()
*-connect	Socket.connect()	TCP.connect()
*-listen	Server.listen()	TCP.listen()
accept	N/A	TCP.accept()	Implicit accept() is called on Server.on('connection')
local-address	Socket.localAddress()	TCP.getsockname()
remote-address	Socket.remoteAddress()	TCP.getpeername()
is-listening	Server.listening	N/A
address-family	Socket.localFamily	TCP.getsockname()
[set-]ipv6-only	N/A	N/A	set via Server.listen() and uv__tcp_bind()
[set-]listen-backlog-size	N/A	N/A	set via Server.listen()
[set-]keep-alive-enabled	Socket.setKeepAlive()	TCP.setKeepAlive()	Calling TCP.setKeepAlive() will set the following socket options: SO_KEEPALIVE=1, TCP_KEEPIDLE=initialDelay, TCP_KEEPCNT=10, TCP_KEEPINTVL=1
[set-]keep-alive-idle-time	N/A	N/A
[set-]keep-alive-interval	N/A	N/A
[set-]keep-alive-count	N/A	N/A
[set-]hop-limit	N/A	N/A
[set-]receive-buffer-size	N/A	TCP.bufferSize()
[set-]send-buffer-size	N/A	TCP.bufferSize()
shutdown	Socket.destroySoon() (?)	TCP.shutdown()

UDP

API	Node.js node:dgram (UDP)	Node.js binding udp_wrap	Notes
*-bind	Socket.bind()	UDP.bind()
stream	Socket.connect()	UDP.connect()
local-address	Socket.address()	UDP.getsockname()
remote-address	Socket.remoteAddress()	UDP.getpeername()
address-family	Socket.address().family	UDP.getsockname().family
[set-]unicast-hop-limit	Socket.setTTL()	UDP.setTTL()
[set-]receive-buffer-size	Socket.[set]RecvBufferSize()	UDP.bufferSize(SO_RCVBUF)
[set-]send-buffer-size	Socket.[set]SendBufferSize()	UDP.bufferSize(SO_SNDBUF)

[guybedford]

I think accept is just a synchronous version of the listen callback, which we can do with the IO thread. And similarly, bind is the listen call itself, without the callback being called.

As such I see no reason not to use the net implementation here, and would recommend we pair on this next week if you have some time then.

[guybedford]

Just to update the report here, on my latest test run, I'm still getting the following test cases failing:

• cli_no_ip_name_lookup
• cli_no_tcp
• cli_no_udp
• preview2_ip_name_lookup
• preview2_tcp_sample_application
• preview2_tcp_sockopts
• preview2_tcp_states
• preview2_udp_bind
• preview2_udp_connect
• preview2_udp_sample_application
• preview2_udp_sockopts
• preview2_udp_states

Any updates very much welcome if I'm missing something.

[guybedford]

I've resolved the three deny tests in #310 - cli_no_ip_name_lookup, cli_no_tcp and cli_no_udp are now passing there