Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SIP-12] Proposal for Branch Management and Release Process #6131

Closed
john-bodley opened this issue Oct 17, 2018 · 14 comments
Closed

[SIP-12] Proposal for Branch Management and Release Process #6131

john-bodley opened this issue Oct 17, 2018 · 14 comments
Labels
sip Superset Improvement Proposal

Comments

@john-bodley
Copy link
Member

john-bodley commented Oct 17, 2018

Motivation

Historically organizations partaking in the release process have managed their own fork of incubator-superset which is deployed in production via a Git submodule. Both companies have been actively developing Superset and deploy internally (features and bug fixes) at a higher cadence than the public PyPI releases. These forks came about because of the potential instability of the master branch (see SIP-13) and provided a mechanism to cherry-pick specific commits thus ensuring an increased stability.

These organizations have discussed ways of trying to fold these forked branches back into the incubator-superset repo to better ensure consistency and to serve as a mechanism for formalizing the release process which is somewhat sporadic.

Proposed Change

The proposed solution is based primarily on having release branches where certain organizations are the primary beta testers for stabilizing the branch.

Master branch

There will be no changes to how the master branch currently works. We will do our best to keep the master branch healthy in part by the initiatives outlined in SIP-13. If you create a pull request, please do it against the master branch. We will maintain stable branches for releases separately but we don’t accept pull requests to them directly. Instead, we will cherry-pick non-breaking changes from master to the two latest releases.

Release Cadence

We propose a regular release cadence of every two weeks. This period is a tradeoff between stability and flexibility ensuring that new features can be deployed within a reasonable time frame. Furthermore the longer the release cadence the greater the potential conflict when cherry-picking commits. A release will be deployed two weeks after it was cut.

We propose that each release branch is fully managed by a single organization.

Branch Naming

We propose that all Git branches follow the <prefix>/<suffix> naming convention:

  • Personal: <username>/<issue>, i.e., jane-doe/issue-123
  • Feature: feature/<name>, i.e., feature/dashboard-v2
  • Release: release/<version>, i.e., release/1.1

Note if / is troublesome we suggest using a two hyphens (--) instead.

Semantic Versioning

Superset follows semantic versioning (<major>.<minor>.<patch>) although the major version has remained fixed at zero. We should release patch versions for bug fixes, minor versions for new features, and the major version for breaking changes. When we make breaking changes, we should also introduce deprecation warnings in a minor version so that our users learn about the upcoming changes and migrate their code in advance.

Git release branches represent the <major>.<minor> level whereas Git tags (associated with a specific commit) represent the fully defined <major>.<minor>.<patch> version. Note tags may contain an optional release candidate suffix (-rc).

The Git Ecosystem

The following diagram describes the apache/incubator-superset Git ecosystem and how we plan to couple this with several organization's internal deployment (superset-internal which uses a Git submodule) which will help serve as a mechanism for stabilizing a release.

release

Day 0

On day 0 the next release branch (release/1.1) will be cut from master which references the latest commit. The commit will be tagged with v1.1.0-rc (the v1.1.0 release candidate) and deployed to PyPI.

Day 0 - 13

Only commits associated which resolve bugs present in the release candidate will be cherry-picked onto the release branch. Fixes to features added after the cut will not be cherry picked (†). Note there may be potential merge conflicts (which will have to be resolved) when cherry-picking a fix due to an upstream commit which is not in the release (‡). If this is too complex the cherry-pick should be skipped (---) as it will be included in the next release. New features can be committed on master if and only if they’re defined in their entirety within the two week cycle. Larger features should be developed on a feature specific branch (which is then merged into master) or using feature flags.

The following diagram shows the use of a feature branch which spans multiple releases.

feature

Day 7

Organizations partaking in the release process will update the Git submodule reference to point to the release (release/1.1). This seven day lag provides a soak period of ensuring that critical bugs are not deployed to our production environments. The update is made to the .gitmodules file,

[submodule "incubator-superset"]
    path = incubator-superset
    url = https://github.com/apache/incubator-superset
    branch = release/1.1

which ensures it tracks the correct release branch. Like the two week release process the submodule branch will be updated every two weeks.

Day 7 - 20

Periodically organizations partaking in the release process will ensure that the submodule in their internal deployment will reference the latest commit from the remote branch (which contains all the cherries) via,

git submodule update --remote --recursive --merge 

which will fetch the latest changes for each submodule, merge them in, and check out the latest revision of the submodule. This step allows us to further test and stabilize the branch in a production environment.

Day 14

The first (non release candidate) version will be tagged (v1.1.0) and deployed to PyPI which references the latest commit on the branch (*). Simultaneously the next release branch (release/1.2) will be cut from master which will mimic v1.1.0 except it will also include any feature commits which were merged in the past 14 days. Note a new release branch is cut if and only if feature commits were added, to prevent duplication.

The following diagram shows an example where on day 14 a new release branch is not cut as there were no new features added since day 0. The next viable release cut date will then be day 28 to preserve the release cadence.

no-release

Day 14 - 27

Additional bug fixes to commits in the release candidate will be cherry-picked onto the n most recent releases (where n = 2, i.e., release/1.1 and release/1.2). Throughout this period the tag can be updated (incrementing the patch) as desired based on the severity of the fix.

Day 28 -

Assuming there is a new release branch (release/1.3) we will no longer be picking cherries onto release/1.1 since we only support two releases (release/1.2 and release/1.3). In summary most release branches will have a four week active lifespan.

Branches should exist for infinitum to ensure critical security fixes can be patched to the necessary branch.

Database Migrations

Database migrations (which are handled via Alembic) have an additional lineage component (used to determine upgrades and downgrades) which further complicates matters. We need to ensure the playback of versions is correct when upgrading to a newer release and that a downgrade is never necessary.

The logic for cherry-picking migrations is similar to above whereas migrations can be cherry-picked onto a release branch if and only if all of its ancestors are present on the branch. Given that there no notion of resolving merge conflicts in Alembic, any migration (and thus future migrations) related to a feature will be skipped (--) and included in the next release.

Action Items

Currently we plan on manually cherry-picking the commits from master to the desired release branch. This could be automated in the future based on PR tags and/or the assistance from a bot.

We also should check with Apache whether there are any concerns/feasibility issues with releasing on a two week cadence.

@beto @mistercrunch

@john-bodley john-bodley added the sip Superset Improvement Proposal label Oct 17, 2018
@mistercrunch
Copy link
Member

A few thoughts:

  • per ASF guidelines, we should not name orgs here (Airbnb/Lyft), but maybe "organizations partaking in the release process"
  • per ASF process, part of the process should involve a [VOTE] thread on the Apache mailing list, perhaps not every release branch needs an Apache release, but every Apache release needs a [VOTE] thread
  • I'm still planning on working on tooling to help with all this mostly around baking release based on a base ref along with Github labels on PR. Cementing the process in this SIP will help dictate what the tooling should do

@john-bodley
Copy link
Member Author

john-bodley commented Oct 18, 2018

@mistercrunch I've removed all reference to specific organizations. I also agree that not every release branch needs to be an Apache release and there's definitely merit in cutting new release branches at a regular cadence.

@mistercrunch
Copy link
Member

Few other thoughts:

  • release and collaborative feature branches are collaborative and history should be preserved (idk if we can enforce this as GH protected branches base on a wildcard release* or something like that)
  • currently we don't use feature/* branches in favor of branches on forks, I think that's common and fine. feature/ branches on the main apache repo would be used when people would need to collaborate on a branch, enforce a merge flow (no rebase or history changes on collaborative branches!) and somehow would need to get rebased / squashed prior to merging on master
  • maybe we need another section on Apache and/or Pypi (convenience) releases.

@mistercrunch
Copy link
Member

Reminder that part of the SIP process should be to notify the Apache Superset dev@ mailing list.

@williaster
Copy link
Contributor

williaster commented Oct 18, 2018

currently we don't use feature/* branches in favor of branches on forks, I think that's common and fine. feature/ branches on the main apache repo would be used when people would need to collaborate on a branch, enforce a merge flow (no rebase or history changes on collaborative branches!) and somehow would need to get rebased / squashed prior to merging on master

Note that we did have a feature branch for dashboard-v2 within the Apache repo pretty much as @john-bodley depicts above^. @graceguo-supercat and I PR-ed into it. We did actually have to re-base somewhat regularly to say in sync with master, but we coordinated a lot about this. I think this worked out okay.

@ShaneCurcuru
Copy link

  • per ASF process, part of the process should involve a [VOTE] thread on the Apache mailing list, perhaps not every release branch needs an Apache release, but every Apache release needs a [VOTE] thread

Every Apache Superset release (i.e. software product released publicly) must have a passing [VOTE] thread on the dev@ (or other appropriate Apache) list. You can certainly manage branches however the PPMC thinks works best but only the PPMC can vote on Apache Superset releases.

As a corollary, other organizations may not release software products that use Apache trademarks. It's not clear from this thread if some of the major contributors here were planning to do their own Superset releases or not, which is why I mention it. Note I'm not involved in the project, so apologies if I'm misunderstanding your process.

@michellethomas
Copy link
Contributor

Superset Deployment Process Update
Sticking to a schedule of a deployment every two weeks has been challenging, and @mistercrunch proposed another schedule that is slightly more flexible and would allow for more time for bug fixing. When we sort out what we need to do to consider these valid releases by ASF standards, we can start trying out this process.

Proposed process:

  1. Always cut on Monday (rc0 and release). If it is a public holiday, postpone to the next business day.
    Take a week for testing and bug fixing
  2. Cherry pick bug fixes for rc1.
    a. Decide if the release is stable enough to deploy. A release would be stable if it has no known major regressions or new bugs blocking critical functionality.
    b. If not:
          i) If there’s something blocking the release that’s easy to revert, we’ll revert it
          ii) Go back to step 2 for rc2
  3. At week 2 if it is not releasable, identify the main blocking issues
  4. At week 3 revert the blockers if not resolved and start a [Vote] email
  5. Tag the release as release 1.1
  6. Continue to fix bugs on this release until the next release goes out.

image

We’ll cut 1.1.0 two weeks after 1.0.0 is cut, and go through the same process. There should not be more than two branches open at a time so 1.2.0 will not be cut until the following Monday after 1.0.0 goes out. If maintaining two branches is too complicated, we may just have one.

A branch shouldn’t take more than 4 weeks to be released because blockers should be reverted at week 3.

The goals of this process

  • Those involved in feature development have enough time to find, and triage issues. Ideally all contributors would be deploying these changes to a test environment and looking for issues.
  • There’s enough time so that whoever introduced the regression can address it. They should be communicating with whoever cut the branch when they expect the bug to be fixed. By default the person cutting the branch will not be responsible for fixing all bugs for a deployment, and contributors should maintain ownership of the successful deployment of their features. If a bug does not get fixed in 4 weeks, the feature will be reverted.
  • There’s a regular structure and clear process so that we all can efficiently and regularly make deployments.

Through this process committers should:

  • Use a “risk:xxx” tag to mark PRs that need thorough testing. Things that may require “risk” tag: db_migration, large feature, or changing large or brittle sections of code.
  • Tag PRs with the release they should go to
  • Tag issues as a bug and which release(s) the bug is in

CC: @john-bodley @kristw

@xtinec
Copy link
Contributor

xtinec commented Dec 20, 2018

Does the new proposal imply we'll attempt to 1) release monthly from master and 2) cut release branches from master every other week?

To clarify, in the diagram/example above, v1.1.0rc2 exists because we reverted certain changes or made bug fixes in the v1.1.0 branch after rc1? If so, do we also apply the same changes in master?

@michellethomas @john-bodley

@DiggidyDave
Copy link
Contributor

DiggidyDave commented Mar 22, 2019

I wanted to chime in here to recommend consideration of Git Flow for the branch/tag/release mechanics. I think it is largely compatible with the broad ideas of SIP-12 described above, but has some benefits:

It also is built around merging and does not include cherry-picking as a matter of course, which keeps history correct and intact. Cherry picks would be rare and used only when absolutely necessary.

A nutshell version of what the above SIP-12 process might look like with gitflow:

Note, gitflow uses a develop branch for baseline development rather than master, and master only contains merge commits from release or hotfix branches. Every commit on master is a tagged release. (see the linked docs above). This is nice because CI can automate release mechanics upon merge to master, knowing it is downstream of all release approvals, testing, voting, etc.

Daily Development

  1. Do feature development against develop branch. Feature, bug-fix, and other "task" branches undergo testing in the feature/task branch for approval, get merged into develop, which should be as close to releasable as possible at all times (it is the new master for CI/CD purposes).
  2. Fix serious non-regression issues (requiring patch in old releases) in the earliest branch that contains them, and merge forward If a bug is found which existed in one of the prior 2 supported releases, and it is serious enough to require a patch release then fix against the first such release, then upward through later previous releases, release branch (if any exist at the moment), then develop.

Releasing

  1. Cut release branch. Lots of feature development, bug fixing etc has been going on and merged into develop. Now we cut a release/1.0.0 branch from develop (which is in an ideal world fairly stable already because adequate testing is occuring prior to approving merges into develop).
  2. Fix release-blockers on release branch. No cherry picking. If a regression or new-feature blocking bug is found it is fixed on the release/1.0.0 branch, then merged into develop. If a non-regression bug is identified (ie one that existed in previous releases), it can be fixed in develop
  3. Tag and publish release candidates as desired. This could be automated to be as often as for each fix on release, if develop is stable enough going into release that we don't have floods of commits going into the release branch. Obvs that's where we would like to be, but easier said than done.
  4. Approve release Do the needful things. Convince selves that it is tested. Bugs are fixed. Votes happen etc.
  5. Release. Merge the release/1.0.0 branch to master, tag the commit v1.0.0, delete the release branch.

A Note about Semver

I just wanted to note that I've had success in the past using a semver scheme where the "version" of a build is automatically generate using a combination of:

  1. a major-minor version declaration in source; I used a version.txt file, for ex.. it must be manual since people building the product are the only ones who know when a change is a major/breaking change or a new feature/minor change (both of which would require an update to version.txt), or just a patch release.
  2. the output of git describe --tags
  3. previous tags on the history

For example, imagine the repo's most recent release tag is v1.2.3 (the release.txt file contained 1.2 for major/minor version), and I cut a branch fixBug1. If I run git describe --tags on that branch with no new commits, it will initially show v1.2.3. If I make one commit, the output would become v1.2.3-1-a83f0c33, which is the most recent tag, the number of commits on top of it, and the sha. Tooling can easily compare that to the major/min version in the file and produce a pre-release semver version string of 1.2.4-fixBug1.1.a83f0c33, which is to say the first commit toward future release 1.2.4 on branch fixBug with a sha of a83f0c33.

If I decided this is a minor version bumper (new feature!), I would modify the release.txt file to contain 1.3. The tooling could notice that 1.3 is a new minor version on top of git's v1.2.3-1-a83f0c33 output and generate a pre-release version string of 1.3.0-fixBug.1.a83f0c33. Likewise, a major version bump reflexted in version.txt could produce 2.0.0-fixBug.1.a83f0c33.

This sort of version string (which is 100% semver compliant) makes it very easy to go from user-facing version strings to the exact source commit that produced it. This is very useful when trying to reproduce bug fixes or confirm that a deployment actually happened, etc.

@kristw
Copy link
Contributor

kristw commented Mar 22, 2019

Hi @DiggidyDave,

Thank you for your comments. When drafting this proposal, @john-bodley, @michellethomas and I did consider gitflow and in some aspects it is similar with the process above. However, there are a few differences:

  • You could consider the current master as equivalent to develop in the gitflow paradigm, and there is no equivalent of master in gitflow. The stable branch is the release branch of the previous version. We adopted this idea from react.js which is also a well-known project. Using master as develop is more suitable with the current workflow of the folks in the Superset community.
  • We also need to maintain multiple versions over time. Now we are maintaining the last 2 minor versions.
    • Version n is the latest release which include all new features since n-1
    • Version n-1 is considered a stable release. That is why we still add bug fixes (via cherrypick) to it to increase its stability and ensure that the previous version is the most stable one we could have.

Oftentimes we do not want to introduce all new features into the current stable release but rather apply bug fixes to improve its stability and wait for the next minor release to adopt new features. In gitflow, once you are at 1.2.3 and create 1.3.0 , it is not natural to create 1.2.4. Only 1.3.1 is possible. Users either have to choose between 1.2.3 which has a bug or 1.3.1 which include the bug fix but also has new features that they are not ready to take. For the multi-version support we adopt some ideas from node.js LTS branches.

Please also keep in mind that Superset is a UI project, in which a new feature that does not break the previous version could still greatly affect the user experience. The choice to upgrade from 1.2.3 to 1.2.4 vs 1.3.x for an organization is not something that can be taken lightly. Sometimes introducing new features require internal communication, etc. to ensure the deployment goes smoothly. Therefore it is important to keep 1.2.x an option when only bug fixes are needed without pushing everyone to adopt 1.3.x mainly for the bug fixes.

Another key point of this SIP-12 is we also wanted to clarify the details about time frame and set the right expectation for releases, not just the git workflow. These details are organization-specific and not specified in gitflow. We research multiple well-known open-source projects such as node.js or Chromium and see how they have regular cadence for releases and adapt them into this SIP.

Again, I appreciate your thoughts and if there are aspects of gitflow that I misunderstood please feel free to discuss more.

References

React.js
Branch Organization
We will do our best to keep the master branch in good shape, with tests passing at all times. But in order to move fast, we will make API changes that your application might not be compatible with. We recommend that you use the latest stable version of React.

If you send a pull request, please do it against the master branch. We maintain stable branches for major versions separately but we don’t accept pull requests to them directly. Instead, we cherry-pick non-breaking changes from master to the latest stable major version.

Semantic Versioning
React follows semantic versioning. We release patch versions for bugfixes, minor versions for new features, and major versions for any breaking changes. When we make breaking changes, we also introduce deprecation warnings in a minor version so that our users learn about the upcoming changes and migrate their code in advance.

We tag every pull request with a label marking whether the change should go in the next patch, minor, or a major version. We release new patch versions every few weeks, minor versions every few months, and major versions one or two times a year.

Every significant change is documented in the changelog file.

node.js
node has minor and patches regularly ~2 weeks
https://github.com/nodejs/node/tags
https://github.com/nodejs/Release#release-plan
https://github.com/nodejs/node/blob/master/COLLABORATOR_GUIDE.md#how-does-lts-work

What is LTS?
Long Term Support (often referred to as LTS) guarantees application developers a 30-month support cycle with specific versions of Node.js.

How does LTS work?
Once a Current branch enters LTS, changes in that branch are limited to bug fixes, security updates, possible npm updates, documentation updates, and certain performance improvements that can be demonstrated to not break existing applications. Semver-minor changes are only permitted if required for bug fixes and then only on a case-by-case basis with LTS WG and possibly Technical Steering Committee (TSC) review. Semver-major changes are permitted only if required for security-related fixes.

Once a Current branch moves into Maintenance mode, only critical bugs, critical security fixes, and documentation updates will be permitted.

Landing semver-minor commits in LTS
The default policy is to not land semver-minor or higher commits in any LTS branch. However, the LTS WG or TSC can evaluate any individual semver-minor commit and decide whether a special exception ought to be made. It is expected that such exceptions would be evaluated, in part, on the scope and impact of the changes on the code, the risk to ecosystem stability incurred by accepting the change, and the expected benefit that landing the commit will have for the ecosystem.

Any Collaborator who feels a semver-minor commit should be landed in an LTS branch should attach the lts-agenda label to the pull request. The LTS WG will discuss the issue and, if necessary, will escalate the issue up to the TSC for further discussion.

How are LTS Branches Managed?
There are multiple LTS branches, e.g. v8.x and v6.x. Each of these is paired with a staging branch: v8.x-staging and v6.x-staging.

As commits land on the master branch, they are cherry-picked back to each staging branch as appropriate. If the commit applies only to the LTS branch, the PR must be opened against the staging branch. Commits are selectively pulled from the staging branch into the LTS branch only when a release is being prepared and may be pulled into the LTS branch in a different order than they were landed in staging.

Any Collaborator may land commits into a staging branch, but only the release team should land commits into the LTS branch while preparing a new LTS release.

Chromium

Chromium has predefined release schedule with certain dates that a release branch will be cut and estimated date that the branch will be stable (~2 months after the branch has been cut).
https://www.chromium.org/developers/calendar

@DiggidyDave
Copy link
Contributor

DiggidyDave commented Mar 22, 2019

Thanks for reply @kristw ... I understood that SIP-12 was largely about release schedule etc, etc., and I think we agree that it is largely separable from the implementation in terms of branch mechanics (ie not specified and needn't be by something like gitflow, which just provides the abstractions for implementing it).

To respond to one point you raised: patching previous releases is very doable in Git Flow. For example, see here: https://stackoverflow.com/a/33052352/3407502 That is what I intended to address in "Daily Development, Item 2" of my comment. Applying the fix on the earliest supported release and merging forward through subsequent supported releases has the additional benefit of completely avoiding cherry-picks and throwing away the history.

Avoiding cherry picks is, IMO, the biggest benefit of switching to GitFlow. Otherwise, much of the react process is transferable... cherry-picking seems like an implementation detail in their model, and it looks like it causes them a lot of trouble, see here for example. I'm actually surprised to see react uses cherry picks so heavily, as it throws away history and context and is prone to producing bugs as you lose track of which branches have changes, where they originated, where subsequent edits were made etc, and with a little bit of rigor they are almost completely avoidable.

A little discipline around release management (ie directing bug PRs towards the correct fix branch) would allow such patches to be smoothly and unidirectionally merged, with history, in a much less risky, chaotic way, to all of the branches, IMO.

Edit: even thought I'm sure y'all have seen these before, I wanted to include a couple of posts from the "please don't cherry pick" viewpoint for posterity :-):
If you cherry pick, your branch model is wrong
Stop cherry-picking, start merging, Part 1: The merge conflict

@kristw
Copy link
Contributor

kristw commented Mar 22, 2019 via email

@DiggidyDave
Copy link
Contributor

DiggidyDave commented Mar 22, 2019

Since people like working against master, let's assume gitflow with master and stable branches instead of develop and master...

Before the hotfix we have:
2 branches, master and stable
many tags: v1.2.0, v1.1.1, v1.1.0, v1.0.0, v0.31.0
we support: v1.2.0, v1.1.1

Summary of process...

  1. Bug is filed, maintainers determine that it is a non-regression that repros back to v1.1.1. So they tag the issue as release:1.1.1 or something to denote that decision.
  2. Bug fix is made on top of v1.1.1 by doing a git checkout -b hotfix-1.1.1 v1.1.1 to create a new branch from the v1.1.1 release. They fix and test the fix in the branch. When it is validated, release votes etc happen if necessary, and the tip of the branch is tagged v1.1.2. The branch can now be deleted (it is no longer needed) and release artifacts can be published from v1.1.2.
  3. Bug fix is merged on top of v1.2.0 by doing a git checkout -b hotfix-1.2.0 v1.2.0 to create a new branch from the v1.2.0 release, followed by a git merge v1.1.1. The fix is tested in the branch. When it is validated, release votes etc happen if necessary, and the tip of the branch is tagged v1.2.1. The branch can now be deleted (it is no longer needed) and release artifacts can be published from v1.2.1.
  4. A PR is created for v1.2.1 into master, and normal PR stuff happens.

After the hotfix we have:
2 branches, master and stable
some new release tags: v1.2.1, v1.2.0, v1.1.2, v1.1.1, v1.1.0, v1.0.0, v0.31.0
we support: v1.2.1, v1.1.2

@rusackas
Copy link
Member

I'm going to go ahead and close this due to age, but @john-bodley please feel free to re-open and kick start the DISCUSS and VOTE threads if you want to rekindle it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sip Superset Improvement Proposal
Projects
Status: Denied / Closed / Discarded
Development

No branches or pull requests

9 participants