diff --git a/README.md b/README.md index 54a7d0e809..fce63e5e14 100644 --- a/README.md +++ b/README.md @@ -35,6 +35,15 @@ A portmanteau word that also puns with "unify", to signify the joining of one co uni - [Latin ūni-, from ūnus, one] FFI - [Abbreviation, Foreign Function Interface] +## Alternative tools + +Other tools we know of which try and solve a similarly shaped problem are: + +* [Diplomat](https://github.com/rust-diplomat/diplomat/) - see our [writeup of + the different approach taken by that tool](docs/diplomat-and-macros.md) + +(Please open a PR if you think other tools should be listed!) + ## Contributing If this tool sounds interesting to you, please help us develop it! You can: @@ -47,34 +56,3 @@ If this tool sounds interesting to you, please help us develop it! You can: ## Code of Conduct This project is governed by Mozilla's [Community Participation Guidelines](./CODE_OF_CONDUCT.md). - ---- - -(Versions `v0.9.0` though `v0.11.0` include a deprecation notice that links to this README. Once those versions have -sufficiently aged out, this section can be removed from the top-level README.) - -### Thread Safety - -It is your responsibility to ensure the structs you expose via UniFFI are -all `Send+Sync`. This will be enforced by the Rust compiler, likely with an -inscrutable error from somewhere in UniFFI's generated Rust code. - -Early versions of this crate automatically wrapped rust structs in a mutex, -thus implicitly making the interfaces thread-safe and safe to be called -over the FFI by any thread. However, in practice we found this to be a -mis-feature, so version 0.7 first introduced the ability for the component -author to opt-out of this implicit wrapping and take care of thread-safety -themselves by adding a `[Threadsafe]` attribute to the interface. - -Version 0.9.0 took this further, and interfaces not marked as `[Threadsafe]` -started issuing a deprecation warning. If you are seeing these deprecation warnings, -you should upgrade your component as soon as possible. For an example of what kind of -effort is required to make your interfaces thread-safe, you might like to see -[this commit](https://github.com/mozilla/uniffi-rs/commit/454dfff6aa560dffad980a9258853108a44d5985) -where we made one the examples thread-safe. - -As of version 0.11.0, all interfaces will be required to be `Send+Sync`, and the -`[Threadsafe]` attribute will be deprecated and ignored. - -See also [adr-0004](https://github.com/mozilla/uniffi-rs/blob/main/docs/adr/0004-only-threadsafe-interfaces.md) -which outlines the reasoning behind this decision. \ No newline at end of file diff --git a/docs/diplomat-and-macros.md b/docs/diplomat-and-macros.md new file mode 100644 index 0000000000..049a81dcd1 --- /dev/null +++ b/docs/diplomat-and-macros.md @@ -0,0 +1,275 @@ +# Comparing UniFFI with Diplomat + +[Diplomat](https://github.com/rust-diplomat/diplomat/) and [UniFFI](https://github.com/mozilla/uniffi-rs/) +are both tools which expose a rust implemented API over an FFI. +At face value, these tools are solving the exact same problem, but their approach +is significantly different. + +This document attempts to describe these different approaches and discuss the pros and cons of each. +It's not going to try and declare one better than the other, but instead just note how they differ. +If you are reading this hoping to find an answer to "what one should I use?", then that's easy - +each tool currently supports a unique set of foreign language bindings, so the tool you should +use is the one that supports the languages you care about! + +Disclaimer: This document was written by one of the UniFFI developers, who has never used +diplomat in anger. Please feel free to open PRs if anything here misrepresents diplomat. + +See also: This document was discussed in [this PR](https://github.com/mozilla/uniffi-rs/pull/1146), +which has some very interesting discussion - indeed, some of the content in this document has +been copy-pasted from that discussion, but there's further detail there which might be +of interest. + +# The type systems + +The key difference between these 2 tools is the "type system". While both are exposing Rust +code (which obviously comes with its own type system), the foreign bindings need to know +lots of details about all the types expressed by the tool. + +For the sake of this document, we will use the term "type universe" to define the set of +all types known by each of the tools. Both of these tools build their own "type universe" then +use that to generate both Rust code and foreign bindings. + +## UniFFI's type universe + +UniFFI's model is to parse an external ffi description from a `.udl` file which describes the +entire "type universe". This type universe is then used to generate both the Rust scaffolding +(on disk as a `.rs` file) and the foreign bindings. + +**What's good about this** is that the entire type system is known when generating both the rust code +and the foreign binding, and is known without parsing any Rust code. This is important because +things like field names and types in structs must be known on both sides of the FFI. + +**What's bad about this** is that the external UDL is very ugly and redundant in terms of the +implemented rust API. + +## Diplomat's type universe + +Diplomat defines its "type universe" (ie, the external ffi) using macros. + +**What's good about this** is that an "ffi module" (and there may be many) defines the canonical API +and it is defined in terms of Rust types - the redundant UDL is removed. +The Rust scaffolding can also be generated by the macros, meaning there are no generated `.rs` +files involved. Types can be shared among any of the ffi modules defined in the project - +for example, [this diplomat ffi module](https://github.com/unicode-org/icu4x/blob/7d9f89fcd7df4567e17ddd8c46810b0db287436a/ffi/diplomat/src/pluralrules.rs#L50-L51) +uses types from a [different ffi module](https://github.com/unicode-org/icu4x/blob/7d9f89fcd7df4567e17ddd8c46810b0db287436a/ffi/diplomat/src/locale.rs#L19). + +Restricting the definition of the FFI to a single module instead of allowing that definition +to appear in any Rust code in the crate also offers better control over the stability of the API, +because where the FFI is defined is constrained. This is +[an explicit design decision](https://github.com/rust-diplomat/diplomat/blob/main/docs/design_doc.md#requirements) of diplomat. + +While the process for defining the type universe is different, the actual in-memory +representation of that type universe isn't radically different from UniFFI - for example, +here's the [definition of a Rust struct](https://github.com/rust-diplomat/diplomat/blob/main/core/src/ast/structs.rs), +and while it is built from a `syn` struct, the final representation is independent of `syn` +and its ast representation. + +## UniFFI's experience with the macro approach. + +Ryan tried this same macro approach for UniFFI in [#416](https://github.com/mozilla/uniffi-rs/pull/416) - +but we struck a limitation in this approach for UniFFI's use-cases - the context in which the +macro runs doesn't know about types defined outside of that macro, which are what we need to +expose. + +### Example of this limitation + +Let's look at diplomat's simple example: + +```rust +#[diplomat::bridge] +mod ffi { + pub struct MyFFIType { + pub a: i32, + pub b: bool, + } + + impl MyFFIType { + pub fn create() -> MyFFIType { ... } + ... + } +} +``` + +This works fine, but starts to come unstuck if you want the types defined somewhere else. In this trivial example, something like: + +```Rust +pub struct MyFFIType { + pub a: i32, + pub b: bool, +} + +#[diplomat::bridge] +mod ffi { + impl MyFFIType { + pub fn create() -> MyFFIType { ... } + ... + } +} +``` + +fails - diplomat can't handle this scenario - in the same way and for the same reasons that Ryan's +[#416](https://github.com/mozilla/uniffi-rs/pull/416) can't - the contents of the struct aren't known. + +From the Rust side of the world, this is probably solvable by sprinkling more macros around - eg, something like: + +```Rust +#[uniffi::magic] +pub struct MyFFIType { + pub a: i32, + pub b: bool, +} +``` + +might be enough for the generation of the Rust scaffolding - in UniFFI's case, all we really need +is an implementation of `uniffi::RustBufferViaFfi` which is easy to derive, and UniFFI can +generate code which assumes that exists much like it does now. +However, the problems are in the foreign bindings, because those foreign bindings do not know +the names and types of the struct elements without re-parsing every bit of Rust code with those +annotations. As discussed below, re-parsing this code might be an option if we help Uniffi to +find it, but asking UniFFI to parse this and all dependent crates to auto-discover them +probably is not going to be viable. + +### Why is this considered a limition for UniFFI but not diplomat? + +As mentioned above, diplomat considers the limitation described above as an intentional design +feature. By limiting where FFI types can be described, there's no risk of changes made "far away" +from the FFI to change the FFI. This was born of experience in tools like `cbindgen`. + +For Uniffi, all use-cases needed by Mozilla don't share this design goal, primarily because the +FFI is the primary consumer of the crate. The Rust API exists purely to service the FFI. It's not +really possible to accidentally change the API, because every API change made will be in service +of exposing that change over the FFI. The test suites written in the foreign languages are +considered canonical. + +## How the type universe is constructed for the macro approach. + +In both diplomat and [#416](https://github.com/mozilla/uniffi-rs/pull/416), the approach taken +is very similar - it takes a path to a the Rust source file/tree, and uses `syn` to locate the special modules (ie, ones annotated with `#[diplomat:bridge]` in the case of diplomat.) + +While some details differ, this is just a matter of implementation - #416 isn't quite as agressive +about consuming the entire crate to find multiple FFI modules (and even then, diplomat doesn't +actually *process* the entire crate, just modules tagged as a bridge), but could easily be +extended to do so. + +But in both cases, for our problematic example above, this process never sees the layout of the +`MyFFIType` struct because it's not inside the processed module, so that layout can't be +communicated to the foreign bindings. +As noted above, this is considered a feature for diplomat, but a limitation for UniFFI. + +This is the problem which caused us to decide to stop working on +[#416](https://github.com/mozilla/uniffi-rs/pull/416) - the current world where the type universe +is described externally doesn't have this limitation - only the UDL file needs to be parsed when +generating the foreign bindings. The application-services team has +concluded that none of our non-trival use-cases for UniFFI could be described using macros, +so supporting both mechanisms is pain for no gain. + +As noted in #416, `wasm-bindgen` has a similarly shaped problem, and solves it by having +the Rust macro arrange for the resulting library to have an extra data section with the +serialized "type universe" - foreign binding generation would then read this information from the +already built binary. This sounds more complex than the UniFFI team has appetite for at +the current time. + +# Looking forward + +## Adapting the diplomat/[#416](https://github.com/mozilla/uniffi-rs/pull/416) model to process the entire crate? + +We noted that diplomat intentionally restricts where the ffi is generated, +whereas UniFFI considers that a limitation - but what if we can teach UniFFI to process +more of the Rust crate? + +It might be reasonable for the foreign bindings to know that Rust "paths" to modules which should +be processed, and inside those modules find structs "marked up" as being used by the FFI. + +In other words, borrowing the example above: + +```Rust +#[uniffi::magic] +pub struct MyFFIType { + pub a: i32, + pub b: bool, +} +``` + +maybe can be made to work, so long as we are happy to help UniFFI discover where such annotations +may exist. + +A complication here is that currently UniFFI allows types defined in external crates, +but that might still be workable - eg, +[diplomat has an issue open to support exactly this](https://github.com/rust-diplomat/diplomat/issues/34) + +## Duplicating structs inside Rust + +When reviewing the draft of this document, @rfk noted that we are already duplicating Rust structs +in UDL and in Rust. So instead of having: + +``` +// In a UDL file: +dictionary MyFFIType { + i32 a; + bool b; +}; +// Then in Rust: +pub struct MyFFIType { + pub a: i32, + pub b: bool, +} +``` + +we could have: +```rust +// In the Rust implementation, in some other module. +pub struct MyFFIType { + pub a: i32, + pub b: bool, +} + +// And to expose it over the FFI: +#[ffi::something] +mod ffi { + + #[ffi::magic_external_type_declaration] + pub struct MyFFIType { + pub a: i32, + pub b: bool, + } + + impl MyFFIType { + pub fn create() -> MyFFIType { ... } + ... + } +} +``` + +So while we haven't exactly reduced the duplication, we have removed the UDL. +We probably also haven't helped with documentation, because the natural location for +the documentation of `MyFFIType` is probably at the *actual* implementation. + +While it might not solve all our problems, it is worthy of serious consideration - fewer problems +is still a worthwhile goal, and needing a UDL file and parser seems like one worth removing. + +## Try and share some definitions with diplomat + +We note above that the type universe described by diplomat is somewhat "leaner" than that +described by UniFFI, but in general they are very similar. Thus, there might be a future where +merging or otherwise creating some interoperability between these type universes might make +sense. + +It seems likely that this would start to add unwelcome constraints - eg, diplomat would not want +its ability to refactor type representations limited by what UniFFI needs. + +However, what you could see happening in the future is UniFFI becoming a kind of higher-level +wrapper around Diplomat. You can imagine a Diplomat backend for UniFFI that converts a .udl file +into a bridge module and then uses the Diplomat toolchain to generate bindings from it, +keeping some of the additional affordances/conveniences UniFFI built for its specific use-cases. + +# Next steps for UniFFI + +As much as some of the UniFFI team dislike the external UDL file, there's no clear path to +moving away from it. We could experiment with some of the options above and see if they are +both viable and worth the investment for the UniFFI use-cases. That sounds like a long-term +goal. + +In the short term, the best we can probably do is to enumerate the perceived problems +with the UDL file and try to make them more ergonomic - for example, avoiding repetition of +`[Throws=SomeError]` would remove alot of noise, and some strategy for generating +documentation might go a long way.