generate UDL from proc macros

[joe-p]

#2081 and general user feedback seems to indicate:

• UDL can be annoying/hard to hand-write
• UDL is useful for readability (by humans or tools)
• Proc macros are a better developer experience

As such, it would be nice to have a solution that enables the upsides of UDL without the sometimes painful requirement of hand writing it. If there was a way to auto generate UDL from proc macros we could have the best of both worlds. Rust developers can use proc macros and in-line documentation as they are implementing the FFI functions while consumers (humans or otherwise) can still have the easy-to-parse UDL file.

It would also be nice to have an API that can be implemented in builds scripts that tracks when the UDL has changes and warn/error on breaking changes.

[mhammond]

Wouldn't we be better off just creating great docs from the procmacros?

[joe-p]

Wouldn't we be better off just creating great docs from the procmacros?

That could be good for human readability but it is still nice to have UDL as input for other tools. For example, from @arg0d in #2081:

We've had very promising results generating C++ scaffolding code to enable C++ libraries to use uniffi. Right now the only way to generate C++ scaffolding is to use UDL. We might investigate some type of proc-macro equivalent mechanism for C++ in the future, but it looks like it would be much more difficult to do than in Rust. The current UDL syntax and lack of informative parsing errors is a big pain point for our C++ teams who have attempted to integrate uniffi into their C++ projects.

I definitely think a docs generator would be useful, but that seems like an orthogonal problem (and one that depends on UDL).

[mhammond]

For input to other tools, wouldn't a format that's machine readable be better? Otherwise these tools would need to take a dependency on our weedle2 fork and duplicate much of the convoluted logic in our uniffi_udl crate to make sense of it.

[joe-p]

UDL already needs to be machine readable and there are currently parsers for it. Sure, there may be problems with the parsers but those are implementation details and not necessarily a problem with UDL itself. As noted in #2081:

The use of a different parser could be implemented as a standalone feature, without revamping UDL syntax.

#2081 highlights two issues:

• The current UDL parser is not user-friendly
• UDL syntax is quite different from Rust syntax

The parser problem can be addressed via a fresh implementation and does not require any breaking changes. The UDL syntax problem is a problem if the expectation is that rust developers are handwriting UDL files, but it seems to me proc macros are preferable to most developers and UDL is being hand-written because it's required by other tools (ie. binding generators).

Auto-generating UDL would be embracing what developers seem to prefer (proc macros) while not making any breaking changes to downstream tooling (binding generators).

If there are fundamental problems with the UDL syntax itself then those should be noted in #2081 but so far it seems like it's mostly an implementation problem.

[mhammond]

#2081 highlights two issues:

As you will see in that issue, I don't fully agree with the characterizations there.

The parser problem can be addressed via a fresh implementation and does not require any breaking changes.

Well sure, but through that lens you could say everything can simply be solved with a fresh implementation.

The UDL syntax problem is a problem if the expectation is that rust developers are handwriting UDL files, but it seems to me proc macros are preferable to most developers and UDL is being hand-written because it's required by other tools (ie. binding generators).

I think this is a misunderstanding of how uniffi hangs together. Zero binding generators consume UDL. Some people do write UDL because it can express things procmacros can not.

Auto-generating UDL would be embracing what developers seem to prefer (proc macros) while not making any breaking changes to downstream tooling (binding generators).

As above, I don't think removing UDL would break any binding generators. The only thing that consumes UDL is uniffi_udl, and that emits types defined in uniffi_meta, which are then consumed by uniffi_bindgen, and that is what bindings consume.

[joe-p]

As you will see in that issue, I don't fully agree with the characterizations there.

Yes and I largely agree with everything you've said. Introducing yet another UDL specification could introduce a lot of problems which is why I'm trying to address the problems outlined in that issue with an outcome that doesn't involve a new specification.

Well sure, but through that lens you could say everything can simply be solved with a fresh implementation.

It's important to separate user-facing file formats from implementations. UDL is a file format that is consumed by various tools. Changing the specification of UDL itself is very different from changing how it is parsed.

I think this is a misunderstanding of how uniffi hangs together. Zero binding generators consume UDL. Some people do write UDL because it can express things procmacros can not.

Nord's c++, C#, and go binding generators all consume UDL.

To be fair, I think you could make a pretty reasonable argument that we could have a specific file format to serve as input for binding generators. For example, serializing the data in uniffi_meta (uneducated guess on where the necessary data currently exists) as some standard format (JSON, TOML, etc.) and recommend binding generators to not directly consume UDL.

"What should external binding generators use as input" is an important question, but they currently use UDL so if we want to change that it should be made clear what the benefits of using a new format over UDL would be. So far the obvious benefits seem to be:

• Allows uniffi-rs to make breaking changes to UDL without affecting downstream consumers (ie. external binding generators)
• If a standard encoding is used, it enables consumers to easily work with the FFI metadata vs having to parse UDL

[mhammond]

Nord's c++, C#, and go binding generators all consume UDL.

I believe they consume it indirectly via how I described it above (ie, they depend on uniffi_bindgen, not weedle2.

"What should external binding generators use as input" is an important question, but they currently use UDL

I don't think that's an accurate representation of the architecture here.

[joe-p]

From the README on how to generate bindings: uniffi-bindgen-cpp path/to/definitions.udl

The UDL file is clearly user input to the tool. If there was a new file format supported by uniffi_bindgen for binding generators then the migration path would be straight forward, but it would be a breaking change nonetheless. Given the small amount of adopted external binding generators it's probably acceptable, but it shouldn't be ignored.

[mhammond]

I'm afraid those docs are slightly misleading - creating bindings directly from the UDL only works in simple single-crate scenarios, but even in those scenarios, the fact the bindings don't directly consume the UDL is relevant to the conversation. Sadly our docs are poor in this regard too. It probably would be possible to have things work from only UDL files in more scenarios, but it's not clear there's actually demand for that - there's certainly none internally.

Given creating bindings directly from the UDL only works in simple scenarios, having something spit out UDL might be interesting for some use-cases, but I don't see those use-cases as ever including feeding that UDL back into the binding generators (or more accurately given the above, back into uniffi_bindgen). It is for this reason that I see no advantage in outputting UDL rather than some other more common machine readable format.

I fear we are starting to talk past each other and I've communicated everything I thought was a misunderstanding, so I'll leave this alone for now.

[joe-p]

Thanks @mhammond. I believe we're on the same page now. I think the main misunderstanding was that I was talking from user perspective and you are talking about how it is implemented. The external generators do not have any code paths for parsing the UDL (as that is handled by uniffi_bindgen), but it is still true that the tools work (in some scenarios) by the user providing a path to a UDL file.

Changes to the UDL seems like it would be a breaking change to binding generators, but that appears to not be the case in most real-world scenarios. This misunderstanding seems like it could be largely resolved via #1241 but I do wonder why parsing UDL is even supported for uniffi_bindgen when it's usefulness is quite limited and introduces this opportunity for breaking changes, albeit insignificant.