Pre-release version numbers

[SimonSapin]

Should Cargo do something special when a dependency has versions published with pre-release version numbers? Maybe not select them unless requested explicitly with cargo update --precise?

CC whitequark/rust-xdg#9, @whitequark, @alexcrichton

[whitequark]

Specifically, if I put xdg = "2.0" or xdg = ">= 2.0" in Cargo.toml, Cargo selects 2.0.0-pre5, which is in clear contradiction of the semver spec, paragraph 11:

When major, minor, and patch are equal, a pre-release version has lower precedence than a normal version. Example: 1.0.0-alpha < 1.0.0.

[steveklabnik]

Yes, Cargo should follow SemVer here.

I think that this might be the fault of the semver crate itself...

[SimonSapin]

Ok, sorting should be fixed, but that’s separate from the original issue.

[whitequark]

Well, given that * is not a valid dependency spec now, what is left in this issue?

[SimonSapin]

Let me rephrase. Let’s say a package has three versions published: 1.0.0, 1.0.1, and 1.1.0-beta. If I depend on it with version requirement 1.0.0, Cargo will currently pick 1.1.0-beta since it’s the latest. But maybe "pre-release" should signal a version that is published for opt-in testing, but is not ready for general use? In that case, Cargo should default to ignore any pre-release version and pick 1.0.1 instead, unless explicitly requested.

Or, more generally, should we assign meaning (and tool behavior) other than the relative ordering to "pre-release"?

[whitequark]

That is exactly what I am speaking about in #2222 (comment)

[SimonSapin]

From the same paragraph that you quoted:

Precedence refers to how versions are compared to each other when ordered.

Precedence is not what I’m talking about.

This part of paragraph 9 is relevant to what I’m talking about, though:

A pre-release version indicates that the version is unstable and might not satisfy the intended compatibility requirements as denoted by its associated normal version.

Should Cargo consider 1.1.0-beta incompatible with 1.0.0 in the same way that 2.0.0 is incompatible with 1.0.0? (Whereas 1.1.0 is compatible with 1.0.0.)

[whitequark]

As far as I'm aware, Cargo's notion of "compatible with x.y" is just >=x.y <x.(y+1), "compatible with x.y.z" is >=x.y.z <x.y.(z+1), and so on for any amount of parts, which is why precedence counts. The part of paragraph 9 you're talking about is illustrative and follows from the precedence rules.

[whitequark]

I.e. since "compatible with 1.0" is >=1.0 <1.1 and 1.1.0-beta < 1.0, 1.1.0-beta will not be considered "compatible with 1.0".

[SimonSapin]

I.e. since "compatible with 1.0" is >=1.0 <1.1

No. It’s >=1.0 <2.0

[SimonSapin]

I.e. since "compatible with 1.0" is >=1.0 <1.1

No. It’s >=1.0 <2.0

Or rather something like >=1.0 <=1.9999.9999 with an infinity of nines, given how pre-releases sorts.

[whitequark]

Er, sorry, yes, you are correct (4AM here...). But my point still stands.

[alexcrichton]

I agree with @SimonSapin that it seems odd if I say "semver compatible with 1.0.0" that I'll start picking up 1.1.0-beta.1 or whatever new prelease becomes available. I think that Cargo may want to specially treat prerelease versions from that form of compatibility, but I think I've also seen some behavior like this in bundler in the past (@wycats perhaps you could clarify?)

I agree that there also may be an issue with the semver crate which needs to be handled as well, especially if we consider 1.0.0-beta to satisfy a requirement for 1.0.0

[wycats]

Two things that I think are somewhat uncontroversial:

• 1.3.0-beta does not supersede 1.2.0
• 1.3.0-beta-2 supersedes 1.3.0-beta-1

I'd like to suggest canonizing the concept of channels, so that:

• 1.3.0-beta-1 supersedes 1.2.0-beta-6
• 1.4.0-alpha-1 does not supersede 1.3.0-beta-6

The idea is to make the Rust-style release cycle more first class and give people a way, through their Cargo.toml, of subscribing to a particular "release channel".

We could also make subscription explicit through additional metadata:

[dependencies.nix]

version = "1.3.0"
channel = "beta"

This would always select the latest betathat also matches the semver versioning (1.5.0-beta-X would match, but 2.0.0-beta-X would not).

[SimonSapin]

Two things that I think are somewhat uncontroversial:

1.3.0-beta does not supersede 1.2.0
1.3.0-beta-2 supersedes 1.3.0-beta-1

What do you mean by supersedes? “cargo update automatically upgrades form one (in your Cargo.lock file) to the other”? If so I agree.

Note that semver.org only talks about precedence, “how versions are compared to each other when ordered”. And 1.3.0-beta does sort after 1.2.0.

I'd like to suggest canonizing the concept of channels, so that:

1.3.0-beta-1 supersedes 1.2.0-beta-6
1.4.0-alpha-1 does not supersede 1.3.0-beta-6

Would there be a list of allowed keywords? semver.org allows arbitrary identifiers for pre-release versions.

[wycats]

What do you mean by supersedes? “cargo update automatically upgrades form one (in your Cargo.lock file) to the other”? If so I agree.

Precisely.

Would there be a list of allowed keywords? semver.org allows arbitrary identifiers for pre-release versions.

Arbitrary identifiers for channels would be allowed, but you could only upgrade across versions on the same "channel".

In practice, it is probably good to stick to a few well-known names like "nightly", "alpha", "beta", "rc", but they would not be interpreted as having any relation to each other across versions.

Just like in Rust, if you subscribe to the "beta" channel, you stick to beta.

[alexcrichton]

One thing we'd need to figure out is what to do when we see a request like:

foo = "1.0.2-beta2"

That's valid by today's rules, but should that only match the package 1.0.2-beta2? Or perhaps 1.0.2-foo would auto-subscribe you to the channel foo?

[jethrogb]

I'm not seeing the behavior that's being talked about in the beginning of the post. In particular, cargo seems to not select prerelease versions at all? Specifying a dependency core_rustc-serialize = "*" (or core_rustc-serialize = "^0.3") results in Cargo choosing 0.3.19, while it should choose the newer 0.3.20-v0.3.19patch1. You can't specify 0.3.20-v0.3.19patch1 in either case by using cargo update --precise. Specifying the exact version string does work. I don't see any code changes linked here that suggest the behavior has changed though.

It seems people in here want cargo to not automatically update to higher prerelease versions, e.g. if the spec is "^0.3.19", it wouldn't update to "0.3.20-alpha". I understand that sentiment, but I think cargo should allow manually updating to that version in this case by using --precise.

[steveklabnik]

cargo seems to not select prerelease versions at all?

In general, it shouldn't select prerelease versions unless you explicitly ask for prerelease versions.

[Boscop]

When I did cargo update, it updated these:

    Updating arithmetic-eval v0.2.0-beta.1 -> v0.2.0
    Updating arithmetic-parser v0.2.0-beta.1 -> v0.2.0

this broke my build!

https://semver.org/#spec-item-11 says:

1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-alpha.beta < 1.0.0-beta < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0

So cargo should not "update" from v0.2.0-beta.1 to v0.2.0.

[Lokathor]

if 0.2.0-beta is less than 0.2.0, can't it update?

[Boscop]

@Lokathor It's less but not semver compatible. It's a breaking change.

[matthiasbeyer]

I just ran into this: I had rustbreak in 2.0.0-rc3 in my Cargo.toml and updated it to 2.0.0. A subsequent cargo check did not pick up this change, I had to run cargo update for it to pick up the change. I would expect otherwise, as this seems not to be semver compatible...

[Nemo157]

@matthiasbeyer that sounds like the opposite of this issue, and a bug if 2.0.0-rc3 was in your Cargo.lock; version requirement 2.0.0 should not match the version 2.0.0-rc3, so the cargo check should have forced an update. Though a quick test shows that it does force an update:

> cargo tree -i rustbreak
    Updating crates.io index
rustbreak v2.0.0-rc3
└── foo v0.1.0 (/tmp/tmp.6sQUs3MEK6/foo)
> cargo add rustbreak --vers 2.0.0
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding rustbreak v2.0.0 to dependencies
> cargo tree -i rustbreak
    Updating crates.io index
rustbreak v2.0.0
└── foo v0.1.0 (/tmp/tmp.6sQUs3MEK6/foo)

[matthiasbeyer]

Well, okay. Maybe messed up somehow in the heat of the moment. Never mind! 😆

[yozhgoor]

I had a problem recently with a published binary (see #9999). I published cargo-temp v0.2.3 using clap 3.0.0-beta.2, Today i was trying to install it via cargo install but it was not possible because i was trying to compile clap with 3.0.0-beta.5, with some breaking change compared to the beta.2.
I use a fixed version =3.0.0-beta.5 to avoid this, but this pitfall can be really silent

[cecton]

I had a problem recently with a published binary (see #9999). I published cargo-temp v0.2.3 using clap 3.0.0-beta.2, Today i was trying to install it via cargo install but it was not possible because i was trying to compile clap with 3.0.0-beta.5, with some breaking change compared to the beta.2. I use a fixed version =3.0.0-beta.5 to avoid this, but this pitfall can be really silent

I just noticed now that leftwm-theme has the exact same issue! https://github.com/leftwm/leftwm-theme

imo this clearly shows that this is a pitfall

[djc]

There was a discussion in the internals forum here: https://internals.rust-lang.org/t/changing-cargo-semver-compatibility-for-pre-releases/14820/10?u=djc.

The preferred suggestion seemed to be this:

maybe an alternative could be that ^1.0.0-beta.1 keeps its current behavior, but 1.0.0-beta.1 changes to be equivalent to =1.0.0-beta.1

Cargo team, does that need an RFC, or just an implementation?

[Eh2406]

I think that will need a detailed description of how people opt in to the breaking change.

[djc]

Okay, here are two directions that we could work out more.

Option 1: reuse the resolver version, bump it to "3" to mean enable sticky pre-releases.

resolver = "3"

Option 2: separate value:

pre-release-updates = "sticky" # or "default"

[Stargateur]

Strike again, rwf2/Rocket#2166, I think a small fix could be that cargo consider by defaut that pre release is for exact version version = "0.1.0-alpha" == version = "=0.1.0-alpha" instead of current default behaviour version = "0.1.0-alpha" == version = "^0.1.0-alpha"

edit: #2222 (comment) I'm blind sorry for spamming.

@djc option 1 seem way better than option 2 cause option 2 require the user to always put this in every cargo.toml file, while "resolver = "3" is a naturel way to just update semver rule, and can be set by default in edition 2024.

[Stargateur]

Rfc proposition rust-lang/rfcs#3263 my first ever ! hope it's ok

[epage]

As this is a more stable place to be holding this conversation, I'm moving a thread over from rust-lang/rfcs#3263

@ehuss said

The Cargo team discussed this RFC, and we think the following may be a viable path forward:

1. Generate a warning if a dependency version has a pre-release version without a SemVer operator. For example, "2.0.0-beta.1" would generate a warning. The warning can be silenced by using an explicit operator like "=2.0.0-beta.1". The warning should recommend the = form. This is to make it a more explicit and conscious choice on how to use pre-releases.

2. In the next Edition, Cargo will require prereleases to have an operator (changes the warning to an error).

There are some risks and drawbacks with this approach that may need some mitigations:

• Users may get stuck in a scenario where they cannot update Cargo.lock because multiple packages are using a shared dependency with = pre-release versions, and they are out of sync. Using pre-release dependencies can be an inherently risky thing to do, so hopefully they are not used pervasively enough that it affects shared dependencies too often. I also feel that usage of pre-releases should be a temporary thing that should hopefully not be used for too long.

• When using = dependencies, users are unable to force an update of a transitive dependency, even when they know it is compatible and safe. The package using the = dependency will need to make a new release to grab any new versions, and this can introduce significant friction and lag that can make using pre-releases frustrating. I suspect it is unlikely Cargo will gain support for some kind of forced-override in the foreseeable future, so this leaves the burden on users to publish new versions to stay up-to-date with new pre-releases (and switching to the final release once it leaves the pre-release stage).

• Updating Cargo.toml to pick up a new prelease can be a tedious process without any sort of tooling support. cargo update currently doesn't provide any help here. I'm personally hopeful that cargo update could gain some kind of flag to also update Cargo.toml as discussed at https://internals.rust-lang.org/t/feedback-on-cargo-upgrade-to-prepare-it-for-merging/17101. This may also require another flag to force the = upgrade.

• This will reduce the visibility of the availability of new pre-releases, which could normally be discovered with cargo update. I'm hopeful that maybe something like cargo outdated could be upstreamed to make it easier to see when newer versions are available.

• cargo add should probably force the use of = (either implicitly add it, or generate an error telling the user to add it) when adding pre-releases.

• We also discussed the possibility of more strongly encouraging users to mark their package as a pre-release if they have any pre-release dependencies. This could be done with a warning message, or possibly a warning during the cargo publish phase, or perhaps just more strongly worded recommendations in the documentation. I'm somewhat uncertain about this particular point, and I'm not sure it is something to gate the RFC on, but I think would be worth mentioning as something to explore (or at least have more discussion here).

@epage said

@ehuss my main concern with requiring an operator is scalability as it can't be automated. Take cargo-release of clap. I can make cargo release automatically add a = on switching to a pre-release but when doing the official release, I have no idea what the intended operator is. clap needs to use = on clap_derive independent of pre-release while I use the implicit operator for everything else. I'd need to go through and individually set the version requirement in each case. This is a 7 crate workspace. I would expect there are larger ones that would like to use pre-release. I feel like this creates enough friction that pre-release will continue to not be practical in cargo.

@ehuss said

Yea, that's another drawback where the switch is lossy. A few thoughts on that:

• If using workspace inheritance, there should only be one place to fix these.

• Presumably publishing pre-releases is a relatively rare event, so I wouldn't expect it to happen too often.

• If automation is important, it may be possible to add some information on how to "revert" the change. For example it could add a comment, maybe something like:

  foo = { path = "foo", version="=2.0.0-alpha.1" } # cargo-release: pre-release-keep-equals

• I think it might be good to more seriously consider the "channels" concept discussed at Pre-release version numbers cargo#2222 (comment) and similar to what is mentioned in the "Future possibilities" in this RFC. That is, a pre-releases will be considered compatible if the first component is the same, but incompatible otherwise (2.0.0-alpha.2 is compatible with 2.0.0-alpha.1, but not 2.0.0-beta.1). That would avoid the need for doing something like =, and provide an avenue for authors to make breaking changes. I think that could still be challenging, since many authors currently treat pre-releases as completely unstable, and won't know about these compatibility requirements which would be somewhat unique to cargo. Perhaps that could be alleviated if cargo gains something like semver checking during publish?

[epage]

@ehuss

If using workspace inheritance, there should only be one place to fix these.

Its number of places and chance for errors (tracking what the requirements should be).

Presumably publishing pre-releases is a relatively rare event, so I wouldn't expect it to happen too often.

Its hard to gauge how much a feature would be used if it had better support. I minimized my use of it for clap v4 and I didn't bother switching my version requirements for the few pre-releases I did do, hoping I wouldn't have a breaking change affected by this, because I didn't want to deal with the version requirement clean up.

If automation is important, it may be possible to add some information on how to "revert" the change. For example it could add a comment, maybe something like:

I hope we can come to a better solution for handling of pre-releases than comment directives.

I think it might be good to more seriously consider the "channels" concept discussed

While channels might be useful in some contexts, I doubt they can generally resolve this problem. A lot of pre-release testing is more one off; people aren't as likely to continually want to do pre-release testing unless they are patching their crates to get ahead of time notice, like people testing nightlies in CI.

That is, a pre-releases will be considered compatible if the first component is the same, but incompatible otherwise (2.0.0-alpha.2 is compatible with 2.0.0-alpha.1, but not 2.0.0-beta.1

imo this is a non-starter. If clap v3 had to have a separate channel for each pre-release breaking change, we would have run out of well-recognized channel names. I had brought this up in one of the other threads that we cannot assume semver compatibility even within the same pre-release type as it doesn't make sense to constrain compatibility on that axis.