Planning for NNG 2.0

[gdamore]

This issue is meant to start the conversation about plans for NNG 2.0.

2.0 is a major, and hence "breaking", release, so applications should expect to need some modification, although we'll try not to make it too painful.

Here are some key thoughts behind the plans -- none of this is committed, it's just thinking at this point!

1. Conversion to meson (CMake will be supported as an output from meson)
2. Rework documentation from man pages into something more natural.
3. Drop support for Windows named pipes - instead relying on AF_UNIX
4. Factor out support for ZeroTier into an external module -- the code needs a complete rework anyway
5. Remove interfaces that are marked deprecated in NNG 1.x.
6. Remove legacy nanomsg compatibility layer (possibly factoring it into an external compatibility shim library) -- wire compatibility still allowed
7. Expose "unsafe" faster APIs to give more performance (e.g. inline aios, etc.)
8. Closing a socket will not destroy it - require applications to call nng_socket_fini()
9. Sockets can become just pointers to structs then (eliminating costly lookups for every operation) -- I plan to expose this as an "unsafe" API in 1.x, possibly.
10. Drop support for Windows versions prior to Windows 10. (So I can adopt newer APIs and AF_UNIX.)
11. Release branch to be separate from 1.x branch (allowing bug fixes to still be applied to 1.x.)
12. Explore new wire protocol to allow lingering close/negotiated close, and negotiated max message. This won't be compatible with legacy nanomsg, but we expect to make similar & compatible changes to mangos.
13. New KCP & QUIC transports (one or both?)
14. UDP transport -- will be unicast only to start, unreliable, and only support small messages (no fragmentation).
15. TLS 1.3 required -- drop support for older TLS versions
16. Better memory API for messages, which should allow loaning/reclaiming application memory
17. Which should facilitate RDMA and possibly shared memory transports
18. User supplied threads, or possibly a callback, to allow easier integration into environments without threading
19. Explore a DPDK port -- depending on 18 above, since we won't have user threads here.
20. Explore picking up the MQTT support by adapting code from nanomq, or writing our own (they have supplied a PR, but for various reasons it never got integrated)
21. APIs for better control over device layers.
22. Prioritization support for pipes (scheduling control...)
23. Some kind of replacement for pair0/poly -- maybe real mesh protocol... (needs support for routing over devices, which means wire protocol changes)
24. Probably all protocols will get a backtrace header (so we never have another pair0/poly situation)
25. Wire protocol version upgrade support (big problem with current design -- no way to negotiate for higher protocols.)

This all looks like a monster amount of effort. Much of it is relatively modest, but some of it cannot be begun without committing to breaking some compatibility. I expect that some of this will be done either in mangos first, or in combination with mangos.

[shikokuchuo]

Fantastic that you have such an ambitious roadmap in mind!

For 12, as discussed previously, it would be useful to be able to wait until a send is complete (not just accepted by the system socket), such that it is safe to reap the socket.

For 22 (I am not sure if this belongs here or separately), it would be useful to have another req/rep semantic where the req node only sends to a rep node for which no reply is currently outstanding, queuing messages at the req socket as necessary. This implements FIFO scheduling for RPC services within NNG itself.

Both of the above would be high impact, allowing simplification of existing solutions.

[gdamore]

The negotiated close is exactly what 12 is about -- it will solve the problem of short lived programs exiting before the other side gets their message.

The idea of using actual outstanding queries for REQ scheduling is interesting (instead of TCP back pressure). The concerns I have about this is that it won't work well across devices, and won't accommodate situations where the cost of the job is not equivalent, and some servers can do a lot of parallelization. Perhaps there could be a way for the servers to provide a response to the submission indicating their ability to accept new work (based on number of contexts they have or something like that?) It will be hard to make that work sensibly in the face of many clients though. I need to think on this.

[shikokuchuo]

You're right, I was thinking only of the simplest case. I had come across this very early issue #10 Perhaps a solution could be to let rep 'workers' request a pipe e.g. when they have at least one outstanding recv.

It is essentially allowing something other than round robin - which is a nice load-balancing heuristic for some types of request, but not particularly suitable for very heterogeneous workloads involving very expensive computations.

[gdamore]

Adding 26: Option setting/getting using untyped option functions will be deprecated or removed. Essentially, if your option is an integer, then you will have to use the xxx_set_int/xxx_get_int forms of these functions. The untyped options that deal with arbitrary blobs of data are just too brittle. (They were modeled on the UNIX setsockopt() call...)

[gdamore]

A bit more about 3: Dropping Windows named pipes will be good, but we probably need to allow for a transition. This will probably involve adding AF_UNIX to NNG 1.x (and also to mangos) so that they can migrate without having to resort to TCP.

[gdamore]

A corollary -- there will be at least one more minor release of NNG 1.x to add AF_UNIX support for Windows. :-)

[JaylinYu]

We have already develoed a QUIC transport layer with msquic. Hopefully 2.0 does not break the classical layering so that our changes are sustained.

[shikokuchuo]

Also just to have this on the record. Hopefully can be a 1.x item.

req/rep when NNG_OPT_REQ_RESENDTIME is set to 0 for the req socket, i.e. no resends, free the msg set on the context i.e. req0_ctx->req_msg early when the send is complete without waiting for the reply.

[gdamore]

Seems reasonable. ;-)

[neachdainn]

Just a few thought, most of the are incredibly minor.

1. Conversion to meson (CMake will be supported as an output from meson)

This will be slightly inconvenient for me, since the CMake integration into cargo is pretty solid whereas the meson integration is mostly just a call out to the shell. But I can add an additional step to the publishing process which, apparently, I only do every year or so, so I'll survive.

2. Rework documentation from man pages into something more natural.

mdBook is very popular in the Rust ecosystem. It might naturally support AsciiDoctor, though I'm not sure.

7. Expose "unsafe" faster APIs to give more performance (e.g. inline aios, etc.)

I assume these will still be covered by SemVer? I would love to have access to these but if they aren't stable then I don't know how much use I can make of them without having to have a Rust wrapper for every single version of NNG.

18. User supplied threads, or possibly a callback, to allow easier integration into environments without threading

These would be somewhere between very and incredibly useful, depending on the exact API. Being able to hook NNG into Tokio (or vice versa) would be awesome.

25. Wire protocol version upgrade support (big problem with current design -- no way to negotiate for higher protocols.)

This, and the other potential changes to the wire protocols, are very interesting and I have a bunch of thoughts on them... If I can remember where I put them. Over the last three years my usage of NNG has been for small, uncomplicated utilities so I haven't had to think about the wire protocols much. Before that, I was making heavy use of NNG in a relatively complicated system and had a bunch of opinions that I'm hoping I wrote down somewhere.

[gdamore]

@neachdainn :

Meson support will not prevent use of the project in CMake friendly projects -- a CMakeLists file is exported by it. So only maintainers of NNG will need to interact with meson. Everyone else will either be able to use CMake like they always did, or can adopt meson if that's more to their liking.

As far as docs, I intend to stick with Asciidoc for now. Markdown simply isn't rich enough. The main changes I have in mind here are in the structural organization of the reference manual - presenting topics that more naturally match expected use, and eliminating the legacy UNIX man(1) sections. This will make presentation more normal for people not coming from UNIX man page thinking, but will make it harder to integrate the documentation in a form accessible via "man". (In fact, I plan to stop supporting the conversion of the content to man format altogether. It's more trouble to maintain than it's worth.)

As far as the unsafe APIs. SemVer will apply to the API, but NOT THE ABI. Meaning, use of unsafe APIs will mean that your application will not necessarily be safe to use across library versions. Indeed, now that I think about this, I might want to only expose these when using a static library build, because they would be inherently unsafe DLLs because I might break those. Or maybe I'll commit to bumping a version if I break that... I need to think on this some more.)

Better support for user supplied threads is definitely going to happen.

Thanks for your thoughts.

[neachdainn]

As far as docs, I intend to stick with Asciidoc for now.

I agree that Asciidoc is much better and it's what I use personally. I just wish it was as supported as Markdown.

SemVer will apply to the API, but NOT THE ABI.

That's definitely workable for my stuff, then. A bit more work but I think that's a reasonable compromise. That being said, I like the idea of bumping versions on ABI changes but I worry that it would be too restrictive on an API that is meant to be closer to the inner workings of the library.

[gdamore]

Actually I looked a bit at mdBook - it looks quite nice, but it is missing some things that I would need to be able to adopt it:

1. Support for admonitions
2. Support for EPUB / PDF backends. (I currently generate publication quality manuals based on this content)
3. Index support (maybe not so strictly required, but I'd miss it for the EPUB/PDF formats.)

[neachdainn]

I was under the impression that mdBook supported 2 and 3, but it turns out those are just community contributions (e.g., mdbook-pdf) and that does seems a bit too risky for a project that doesn't want to constantly rework documentation.

EDIT: All of the official unofficial backends, including PDF and EPUB. The official unofficial plugins also include support for admonitions but, again, 3rd party might not be the way to go.

[stephanlachnit]

#1473 would be a nice addition to the wishlist

[gdamore]

As a heads up, the dev2.0 branch now contains the start of this work.

[shikokuchuo]

HTTP/2 with support for SSE would be nice.