Combating version churn

[Jake-Shadle]

First of all I just want to thank the maintainers/Microsoft for making these crates, having a set of maintained bindings that can supplant winapi and be confident will be maintained in the future has been really great.

Problem

That being said, I think the current velocity of version changes in this repo is a bit problematic for the ecosystem, especially as more crates adopt windows or windows-sys over winapi. This is because, for the past several months, the crates in this repo have have had their minor version bumped on average every couple of weeks, essentially guaranteeing that with enough transitive dependencies on one of the crates in this repo, a project's crate graph will include multiple versions of one or more windows crates. Duplicate versions can be fine for short periods of time as various crates catch up to newer releases, however the velocity of this repo is far greater than many crates in the ecosystem, especially in the windows-sys crate case of being low level system bindings.

For a simple example, let's take parking_lot.

• parking_lot currently depends on windows-sys:0.34, which is around 1.5 months old
• Someone helpfully added a PR to bump to windows-sys:0.35 (22 days old) about a week ago
• windows-sys was bumped to 0.36 a few hours ago

parking_lot is now in a conundrum. It could accept the 0.35 PR, and cut a release, but there's already a newer version, so they could bump directly to 0.36 instead, skipping 0.35 entirely. However, depending on what other crates you have in your graph, either decision is (or the none option of just staying on 0.34) almost guaranteed to result in multiple versions of windows-sys as

• Some crates will have transitioned to 0.35 sometime since it was released, but might take several days/weeks/months to move to 0.36
• Some crates will be on 0.34 (or even older) versions like parking_lot
• Some crates might move to 0.36 quite quickly, which could mean you're depending on up to 3! different versions of windows-sys simultaneously

The real kicker is that parking_lot uses a grand total of 3 function bindings and a few type/constant bindings which, beyond fundamental changes such as #1439, are...."never" going to change, thanks to Windows' backwards compatibility guarantees.

If we take a step back I see there being multiple reasons for this version churn

• All (or at least all the "user facing" ones) of crates in this repo share a single version. On the one hand this is nice since versioning is internally consistent and makes perusing version history more easy etc. However, it's also a very big hammer, particularly in the case of windows-sys, as it means that any change in any crate in this repo, regardless of the downstream usage, results in a new version.
• The Windows API surface is positively massive, and due to how the windows/-sys crates are organized via feature flags, any breaking change to one sub-API is technically a breaking change to the crate as a whole. This is despite the fact that an overwhelming majority of downstream users will use an incredibly small fraction of that massive API surface, meaning (again, other than fundamental changes such as Don't specialize HANDLE #1439) that they're very unlikely to actually observe such a breaking change in practice.
• Reiterating from above, the release cadence of this repo is just much higher than most other crates in the ecosystem, but because the bindings in windows/-sys are essentially the libc of Windows, many crates depend (or will, once they transition from eg. winapi) on them which coupled together mean multiple versions of one or more crates in this repo become almost inevitable

Possible Solutions

Below are several options that I can see that could solve, or at least alleviate, this problem. Obviously it's up to the maintainers on what they think is best (or if it will be considered a problem at all), so these are just some thoughts.

• Stabilize and use patches. The crates in this repo are still early days so I doubt this could happen anytime soon, but having a model similar to libc, where they've been on a 0.2 minor version for years and only ever bump the patch version, means semver compatibility across the ecosystem. libc obviously has a bit of a simpler problem since the API surface is orders of magnitude smaller than Windows, but it is a proven model.
• windows-sys in particular (maybe not windows?) could be more lenient about breaking semver. What I mean be this is, in an overwhelming majority of use cases, a crate using windows-sys will use an incredibly small fraction of the API surface exposed by it. This means that (again, beyond fundamental changes like Don't specialize HANDLE #1439) that a breaking change in windows-sys, such as a moving constants or functions to a different location, are not going to be observed by the downstream crate at all unless they were using those exact functions/constants. Speaking for myself, this kind of breakage would be entirely acceptable if it was clearly stated in the release notes, or even just as a general caveat clearly stated in the README as it is trivial to find where a function/constant was moved to in a newer version and change it to the new location/feature flag. This is unfortunately not a super viable method for transitive dependencies however, I just thought I would mention it.
• Make windows-bindgen a first class citizen (related windows-bindgen needs documentation #1407). At least in my use cases, and I believe many others, using Windows bindings is a fairly "write once" operation that generally doesn't change much past the initial implementation. For this kind of use case, having a convenient way to do "generate functions x, y, and z, and these constants to this output file" would be frankly amazing. For example, here's a recent PR I made to add minidump writing. In grand total, this uses 5 functions, ~6 constants, and ~6 types. These will most likely never change again, and by generating only the bindings I need, once, and just including them as a single source file directly in the project, I could get rid of the windows-sys dependency altogether, avoiding the problem of multiple versions now and in the future. If we go back to the parking_lot case above, it's already manually creating its own bindings to Nt* functions that are not part of Win32 metadata, and could do the same for the few functions/constants that it is using from windows-sys, but having that functionality exposed in a friendly way by a windows-bindgen tool would mean generating the bindings once and never having to worry about version churn in the future, because again, these functions/constants/types are not going to change

Closing

I'm sure the maintainers have probably thought about this issue already, but I'm opening this issue because I couldn't find anything relevant that was already open and want to foster a discussion in the open about this, since I'm sure we're not the only ones that find the current situation problematic.

Sorry this was a bit of a braindump I've just been thinking about this the last few days and wanted to finally just write it down.

[ChrisDenton]

I would also love to see a solution here. Of the options currently suggested, personally I think the third would be the nicest of those given for reasons you've already brought up. So I'll just summarize my thoughts here.

I'm not sure that libc is a good role model for these crates at this time. As you say, it's effectively stuck on 0.2 because of fears of another "libcpocalypse", where bumping the version from 0.1 to 0.2 broke the world. Eternal 0.2 may be manageable for libc but both windows and windows-sys are currently under much more active development. More worryingly, the upstream metadata is still tagging releases as a "preview" and makes frequent changes to fix both major and minor issues.

I would also not be keen on breaking semver as tooling does rely on it, although I do agree that any individual crate is unlikely to break on updates too often unless they happen to be one of the unlucky ones.

Which leaves making windows-bindgen a first class citizen. I think this is the best option because it leaves windows and windows-sys free to evolve while giving crates a way to opt-out of the, currently frequent, upgrade cycle.

[poliorcetics]

using Windows bindings is a fairly "write once" operation that generally doesn't change much past the initial implementation

Note this only really works for private implementation details. If anything is exposed through the public API of a crate, I don't want to have to convert between each crate that implements the same type in its own bindings all the time, I want them to use windows(-sys).

That's something the windows-bindgen crate would have to document very thouroughly to avoid very painful compatibility problems down the line

[kennykerr]

Great issue! We are indeed aware of it, but an obvious solution is elusive. I will re-read the comments thus far and add any of my own. Keep it coming - I'm confident that together we'll come up with a good solution in the end.

[rylev]

Thanks @Jake-Shadle for the write up! We have indeed been thinking about this, but we've yet to come up with a good solution. Let me try to talk through the considerations (many of which you've already touched on) so we're sure we're on the same page. Hopefully this makes it easier to talk through possible solutions.

• windows-sys is basically a raw projection of the win32metadata project into Rust.

  • That project is itself relatively new and therefore far from being able to declare stability guarantees. I will note that we hope to work closely with that project on ways to think about stability not as either 100% stable or 100% unstable. Stay tuned...
  • windows-sys has effectively no internal logic (i.e., it is essentially a 1 to 1 mapping of the metadata), so the stability of windows-sys is 100% tied to the stability of win32metadata.

• A single monolith solution broken up by features (i.e., the status quo) means that changes to any part of the win32 projection will result in version bumps, even if the particular piece of the Windows API a downstream crate is using does not change.

  • This leads to fracturing of the ecosystem like you mentioned in your post above with parking_lot and what libc ran into.
  • What's more any changes to the metadata will likely result in a breaking change to the public API and thus, under strict semver, require a major release. You mention the possibility of playing fast and loose with semver, but we'd prefer a solution that plays by the rules if possible.

• A multi-crate solution (i.e., breaking up windows-sys into more manageable chunks that can evolve independently) is difficult. Due to its long history, the Windows API is not that well factored and there are many cyclic dependencies between different pieces of the API. This is something that the crates model does not allow for - for good reason.

  • It certainly seems possible to break up the surface in some way, but whether that leads to a solution that addresses the issues you laid out in a sufficient way, remains to be seen.

• Providing code generation tooling is problematic. It is much more preferable to provide a crates.io based solution for all the same reasons that Rust prefers declarative crate dependencies rather than copy/pasting source code. The latter makes providing bug fixes, security patches, code reuse, etc. difficult and is an antipattern we'd like to avoid if possible.

  • And as @poliorcetics pointed out this can also easily lead to fractured ecosystem issues if folks start using such a solution in any publicly visible way.

My personal opinion is that a multi-crate solution (if possible) would be the right way to go. This is essentially the solution that Cargo and crates.io push us towards. It also makes other issues we've run into go away (e.g., rustdoc struggles with such a massive monolithic crate). However, the Windows API surface does not make this easy, so we still have to do work to even understand how we would begin to break up windows-sys.

[kennykerr]

This was also previously discussed here: #432

[kennykerr]

In the short term, I'll probably exclude windows-sys from the regular 3-week release cadence that we've been on to allow the current version (0.36) to establish itself. Notably, windows-sys 0.36 has now been adopted by both tokio and parking_lot.

[bjorn3]

A multi-crate solution (i.e., breaking up windows-sys into more manageable chunks that can evolve independently) is difficult. Due to its long history, the Windows API is not that well factored and there are many cyclic dependencies between different pieces of the API. This is something that the crates model does not allow for - for good reason.

Would having one crate for all type definitions and then several for function declarations work? Function declarations can't have cyclic dependencies. Or do the type definitions also regularly change?

[rylev]

Would having one crate for all type definitions and then several for function declarations work?

I'm not sure that the developer experience hit would be worth it. Normally folks tend to think of APIs as functions and types bundled under some logical namespace. For example, if you're making an app that relies on direct2d, it feels natural if all of the APIs that logically fall under the direct2d banner are in 1 dependency. Having to break up by type vs function might be confusing, and I don't think it wins us a whole lot. I'll think about this more.

[bjorn3]

The types can be re-exported by the individual function crates.

[kennykerr]

A huge portion of the Windows API is defined in terms of COM and WinRT types and those don't really make sense to split up in that way either.

[rylev]

FYI: I have introduced an RFC mechanism in #1731. At some point, when someone feels they have a solid understanding of the problem and a solution that addresses the concerns outlined in this issue, it would be good to open an RFC proposing how to address this. I know, for example, some folks at Microsoft are investigating the feasibility of a multi-crate approach.

[kennykerr]

Just a quick update on this issue. While I've been exploring a multi-crate solution, it is far from a home run.

I have excluded the windows-sys crate from the regular release cadence which seems to largely deal with the version churn in practical terms. For the foreseeable future, we will release updates to the windows-sys crate only when there is sufficient demand.

We are also doing a lot of work to reduce the overall size of the windows crate and it too will get to a point where it needs few if any updates and will also be able to move into a demand-driven release model.

[kennykerr]

I also wanted to mention that windows-bindgen should continue to improve and stabilize as an alternative to the pre-generated windows and windows-sys crates for even more demanding scenarios. We continue to work hard on improving Windows support but I'm still holding out hope that Rust itself can improve to make this less of a problem, as @joshtriplett mentioned here: #1285 (comment)

Anyway, I'll close this issue for now as there isn't really any actionable work here. Feel free to keep the conversation going and if new information comes to light, we can always revisit this topic.

[rylev]

I don't really see any reasoning here for why a multi-crate solution is not viable. All that was written is " While I've been exploring a multi-crate solution, it is far from a home run." What about it was not a homerun? It would be really great to get a breakdown of the pros and cons of each approach so others can get insight into the decision-making process here.

[kennykerr]

I don't really see any reasoning here for why a multi-crate solution is not viable.

I'm not necessarily saying it's not viable, although I have my doubts. I'm saying the original issue - version churn of the windows-sys crate - has been addressed. There were many other ideas floated here and we may continue to explore some of them. The multi-crate idea in particular is one I spent at least two weeks exploring but didn't arrive at a promising proof of concept. I may continue to explore that idea as time allows.

[rylev]

The multi-crate idea in particular is one I spent at least two weeks exploring but didn't arrive at a promising proof of concept.

This is exactly what I mean - it would be really nice to know the issues that were run into. Perhaps others have ideas on how to overcome the limitations you've found. Writing up your findings gives others a way to check your assumptions and assist in finding solutions.

[kennykerr]

Agreed and still plan to do that when I get a moment, probably using the RFC model.

[kennykerr]

Here you go: #2396

Let me know if this addresses the need.

[kennykerr]

@Jake-Shadle would you be willing to update your PRs to use #2396?

[Jake-Shadle]

Sure.

[kennykerr]

Thanks all!

https://crates.io/crates/windows-bindgen v0.47.0 has now been released and includes the new standalone code generation support.

[glandium]

Briefly, I spent a few weeks trying to make it work but the circular nature of many API definitions collides with Cargo's restriction on circular dependencies and even when I artificially broke those cycles, the transitive tendency of APIs combined with crates and features meant that you ended up spending more time compiling rather than less and resulting in a far more complex model for specifying dependencies than merely using Cargo features.

I looked at this recently-ish, and while I haven't gone as far as implementing something concrete, it seemed, at least for windows-sys (not sure about windows, I haven't dug deep enough in what its APIs look like), that splitting types and functions would solve the circular dependency problem. Is that something you considered?

[kennykerr]

Yes, the challenge is with splitting in a way that (a) solves the toolchain problems and (b) continues to be easy to use and maintain. With over 600 namespaces defined in metadata, that turns out to be challenging technically and practically.

Anyway, we now have three viable ways to consume Windows APIs namely the windows crates, the windows-sys crate, and the windows-bindgen crate:

• https://kennykerr.ca/rust-getting-started/windows-or-windows-sys.html
• https://kennykerr.ca/rust-getting-started/standalone-code-generation.html

I don't think even more options would be beneficial. My focus for this year remains on metadata stabilization and tooling so that we can reign in the metadata and more easily reshape and balance the definitions in a way that would be more beneficial to Rust and allow us to finally stabilize the windows and windows-sys crates and avoid further breaking changes.

[MarijnS95]

@kennykerr is standalone bindings generation for the windows (not windows-sys) crate, with COM support and high level bindings, also still on your radar for this year? I feel that will also address these "splitting" issues for the majority of the crates that only need to bind to Windows, but don't have any such types in their public API (https://github.com/Traverse-Research/gpu-allocator does though, unfortunately - and it needs D3D12 COM access).

[kennykerr]

That is a much bigger task and something I'm working on, but I'd like to see how well the existing standalone support is received and adopted before adding support for the windows style bindings.

I've fixed all reported issues to improve the usability of this feature and I hope to see this help the Rust community improve support for Windows (#2416 #2421 #2426 #2430).

[kennykerr]

@Jake-Shadle let me know if there's anything else you need to wrap up parking_lot_core - that's the only one that currently seems to be in limbo - happy to publish an update to the windows-bindgen crate if the repo has everything you need.

[kennykerr]

FYI Amanieu/parking_lot#378

[Ralith]

Today this is geared towards producing bindings for whole namespaces, but this could be extended to support discrete APIs if needed as well.

I almost invariably need only a few definitions at a time (e.g. in quinn-udp), so this is appealing.

[kennykerr]

Just a quick update for those still following along. I'm nearly ready to publish the windows-core crate that takes the foundational support in the windows crate's core module and splits it out into a dedicated crate so that it may be used without the hefty windows crate and then support standalone code generation of the windows-style bindings.

[kennykerr]

The windows-core crate is now ready for testing, along with standalone code generation support for COM: #2475

[MarijnS95]

@kennykerr thanks, this is looking very promising already! While in the process of converting one of my apps to use these minimal COM bindings, I ran into not getting the From<> binding from crates/libs/bindgen/src/extensions/mod/Win32/Foundation/BOOL.rs: how do I get that with standalone_win()?

Also, #2475 doesn't explain what and how to use standalone_std: documentation seems to indicate that it's for internal Rust STD library use?

[kennykerr]

Thanks for kicking the tires - please create new issues for any problems you discover - that way I won't forget to address them.

Yes, standalone_std is only for the Rust Standard Library's internal use. I'll update the following with more info:

https://kennykerr.ca/rust-getting-started/standalone-code-generation.html

[MarijnS95]

@kennykerr thanks, I'll file a separate issue for the lack of extensions soon. Working code here: MarijnS95@e8f3665

Perhaps standalone_std could be renamed? It initially provokes the assumption that it'll provide std-based implementations (and the rest would purely be Windows/core).

[ChrisDenton]

Hmm, maybe it should have a #[doc(hidden)] on it?

[kennykerr]

Yes, I'll just hide it.