Adding a minor version number

[zanieb]

Hi everyone!

We are interested including an additional version number (the minor version) in our versioning scheme with the intent of alleviating some of the problems with the current 0.0.x versioning scheme. Following is a chunk of an internal RFC proposing the change. We're excited to get your feedback!

Introduction

There are a few problems with our current versioning scheme:

• There is no documented policy and little room for a meaningful policy
• Rules are added in most releases, resulting in significant new violations for users on upgrade
• Users cannot get bug fix upgrades with confidence that they will not need to do other work
• Organizations do not perceive the tool as stable, preventing adoption

We have the following goals for adding a minor version number:

• Reduce the frequency of large changes
• Give rules a period for stabilization before
• Publish a versioning policy for users to reference
• Indicate movement towards stability
• Differentiate between breaking and non-breaking releases

The proposed versioning scheme can be summarized as follows:

• We will not use the major version at all in this phase of the project.
• The minor version will be used for stabilization of preview features, changes to stable behavior, and breaking changes.
• The patch version will be used for bug fixes and any changes to rules or fixes in preview.

In the following sections, we'll cover concrete cases of when version numbers will change, introduce a preview mode, and discuss stabilization of rules and fixes.

Version changes

Minor version increases will occur when:

• A deprecated option or feature is removed
  • The length of the period discretionary, with regards to the maintenance cost of keeping the option or feature.
• Configuration changes in a backwards incompatible way
  • This may occur in minor version changes until 1.0.0, however it should generally be avoided.
• A rule is promoted to stable
• Support for a new file type is promoted to stable
• Support for an end-of-life Python version is dropped
• The behavior of a stable rule is changed
  • Does not include bug fixes that follow the original intent of the rule
• Stable rules are added to the default set
• Stable rules are removed from the default set
• An automatic fix for a rule is promoted to stable

Patch version increases will occur when:

• Bugs are fixed, including behavior changes that fix bugs
• A suggested or manual fix for a rule is added
• A automatic fix for a rule is added in preview
• A fix’s applicability is demoted
• A new configuration option is added
• A rule is added in preview
• The behavior of a preview rule is changed
• Support for a new Python version is added
• Support for a new file type is added in preview
• An option or feature is deprecated

Note that these lists are not going to be the same once we add a major version number. For example, we are currently including the addition of features in the patch version to avoid signaling breaking changes but we will likely follow semantic versioning and use the minor version for such changes in the future.

Preview mode

A --preview flag will be available to enable new, unstable rules and features e.g. support for a new file type.

The preview mode is intended to help us collect community feedback and gain confidence that changes are a net-benefit.

The preview mode is not intended to gate access to work that is incomplete or features that we are likely to remove. Preview mode should not add or remove CLI options.

Despite the intent of preview mode, we reserve the right to make changes to any behavior gated by the mode including the removal of preview features or rules.

Rule stabilization

When modifying or adding rules, we will use the following guidelines:

• New rules will always be added in a preview mode
• New rules will remain in preview mode for at least one minor release before being promoted to stable
  • If added in a patch release i.e. 0.6.1 then a rule will not be eligible for stability until 0.8.0
• Stable rule behavior will not change significantly in patch versions
• Promotion of rules to stable may be delayed in order to “batch” them into a single minor release
• Not all rules in preview need to be promoted in a given minor release

Fix stabilization

Fixes have three applicability levels:

• Manual: Never applied by Ruff, just displayed.
• Suggested: Suggested by Ruff, applied with opt-in.
• Automatic: Applied by Ruff when fixing is enabled.

Fixes for rules may be introduced at a lower applicability then promoted to a higher applicability. The applicability of a fix may also change when the preview mode is enabled.

Conclusion

Users will be able to upgrade between patch releases without inheriting behavior changes. At the same time, rules will be phased in via preview mode to allow for stabilization of new features over the course of a few releases.

While consensus on this document is a prerequisite for v0.1.0, there are other things we'll need to do once that is achieved. For example, we'll need to implement tooling to support preview mode and update the CLI to respect fix applicability.

[leandrodamascena]

Hi @zanieb and the Ruff team! This is completely amazing. We now have a clear versioning scheme and we know when we can update Ruff in our pipelines and avoid breaking anything.

I think the RFC covers what needs to be defined in each new release. I missed the criteria in which a rule will be promoted to stable. I understand it must be promoted with at least 2 minor releases in preview, but what are the criteria for a rule to leave preview and become stable? Community feedback? Bugs reported? Adoption of that rule (how do you know?)? Do you think it's worth exploring more this question? Sorry if I missed something, but it's not clear to me.

Thank you so much for giving customers even more confidence in the Ruff project. This is certainly a significant change in this project.

[zanieb]

Thanks for your comments!

What are the criteria for a rule to leave preview?

This is a great question and is not yet something that I have a strong opinion on. I think the criteria is roughly that we have confidence the rule will:

• Not cause panics / internal errors
• Not break code on fix
• Have a reasonable number of false positives and negatives
• Be seen positively by the community

I think that it's hard for us to include in the policy, in part, because it will often depend on the rule in question. For example, Ruff may not yet have the capabilities that we need to enforce the rule consistently and we may choose to wait for those capabilities to be added before stabilizing the rule.

It's also likely that our criteria changes over time as we learn from promoting rules. At this point, we have no idea how many users will try a preview mode and how comprehensive testing will be. In this RFC, we're focused on improving the current situation and I hope by doing so we'll have the opportunity for perspective on what kind of future improvements we can make.

[leandrodamascena]

Hey @zanieb! I agree it is difficult to define the details of how to promote a rule in preview state to stable and I don't want to make that a potential problem in this RFC.

Thank you for making your comments and I am available for further discussions.

[zanieb]

Now that we've announced our formatter I'd like to add the following proposals for formatter versioning that would take effect once it reaches stable:

Minor version bumps when:

• The stable formatting style is changed

Patch version bumps when:

• The preview formatting style is changed
• A bug in the stable or preview formatting is fixed

Here it's important to distinguish between changes in formatting due to bugs versus changes in formatting style. A formatting bug fix may address:

• An edge-case in an existing stylistic choice — i.e. the change must be consistent with the intent of the existing style.
• Comment placement that breaks other tools or is inconsistent with our general rules for comments.
• Invalid syntax after formatting
• Unintentional deviations from standards e.g. Black

In contrast, formatting style changes are subjective attempts to improve the readability of the code.

Discussion welcome!

[hugovk]

Thank you for putting all this together!

This seems a bit like semantic versioning, except keeping major = 0 and using minor for both breaking changes and new features. (Patch remains as patch for bug fixes.)

May I ask why the hesitation to use major version numbers?

Rather than coming up with a new versioning scheme, it seems to me it would be a lot simpler to bump to 1.x and follow SemVer, which says:

How do I know when to release 1.0.0?
[1] If your software is being used in production, it should probably already be 1.0.0. [2] If you have a stable API on which users have come to depend, you should be 1.0.0. [3] If you’re worrying a lot about backward compatibility, you should probably already be 1.0.0.

I believe those are all met:

1. Yes, Ruff is wildly successful and widely used in production. It was downloaded 32 5 million times last month!
2. Ruff may not have an API as such, but users have come to depend on the lint rules being stable.
3. This discussion shows you do care about users, are worrying about backwards compatibility, and do want a clear policy (and thank you!)

Applying SemVer to the points in the first post could give something like this [my notes in italics]:

Major version increases will occur when:

• A deprecated option or feature is removed
  • The length of the period discretionary, with regards to the maintenance cost of keeping the option or feature.
• Configuration changes in a backwards incompatible way
  • This may occur in minor version changes until 1.0.0, however it should generally be avoided.
• The behavior of a stable rule is changed [this could possibly remain minor]
  • Does not include bug fixes that follow the original intent of the rule

Minor version increases will occur when:

• A rule is promoted to stable
• Support for a new file type is promoted to stable
• Support for an end-of-life Python version is dropped [could be major, but Requires-Python metadata takes care of this]
• Stable rules are added to the default set
• Stable rules are removed from the default set
• An automatic fix for a rule is promoted to stable
  [the below are promoted from patch because they're new features, not bug fixes]
• A suggested or manual fix for a rule is added
• A automatic fix for a rule is added in preview
• A fix’s applicability is demoted
• A new configuration option is added
• A rule is added in preview
• The behavior of a preview rule is changed
• Support for a new Python version is added
• Support for a new file type is added in preview
• An option or feature is deprecated

Patch version increases will occur when:

• Bugs are fixed, including behavior changes that fix bugs

[zanieb]

Hi thanks for your comment and notes!

While many people are excited about Ruff, I think we want to focus more on building an amazing tool and less on versioning and backwards compatibility. Waiting to add a major version number (and strictly following semantic versioning) helps us manage expectations and allocate our teams resources properly.

This discussion is a big step towards a major version number — we're already working on a versioning scheme for when a major version number is added. Adding a minor version number first will allow us to incrementally explore and test our versioning policy with feedback from our users. We hope this will guide us towards a dependable 1.0 release.

I know you've already participated in #1992 but hopefully the discussion there gives you (and other readers) an idea of the kind of projects we want to work on to define Ruff's stable API.

p.s. we're curious where the "downloaded 32 million times last month" statistic come from, it looks more like 5 million?

[hugovk]

Oh sorry, 32 million was for another package. But 5 million is still exceptional!

[zanieb]

Ah okay haha we thought maybe it was all-time or something :)