-
Notifications
You must be signed in to change notification settings - Fork 14.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SIP-12] Proposal for Branch Management and Release Process #6131
Comments
A few thoughts:
|
@mistercrunch I've removed all reference to specific organizations. I also agree that not every release branch needs to be an Apache release and there's definitely merit in cutting new release branches at a regular cadence. |
Few other thoughts:
|
Reminder that part of the SIP process should be to notify the Apache Superset dev@ mailing list. |
Note that we did have a feature branch for |
Every Apache Superset release (i.e. software product released publicly) must have a passing [VOTE] thread on the dev@ (or other appropriate Apache) list. You can certainly manage branches however the PPMC thinks works best but only the PPMC can vote on Apache Superset releases. As a corollary, other organizations may not release software products that use Apache trademarks. It's not clear from this thread if some of the major contributors here were planning to do their own Superset releases or not, which is why I mention it. Note I'm not involved in the project, so apologies if I'm misunderstanding your process. |
Superset Deployment Process Update Proposed process:
We’ll cut 1.1.0 two weeks after 1.0.0 is cut, and go through the same process. There should not be more than two branches open at a time so 1.2.0 will not be cut until the following Monday after 1.0.0 goes out. If maintaining two branches is too complicated, we may just have one. A branch shouldn’t take more than 4 weeks to be released because blockers should be reverted at week 3. The goals of this process
Through this process committers should:
CC: @john-bodley @kristw |
Does the new proposal imply we'll attempt to 1) release monthly from master and 2) cut release branches from master every other week? To clarify, in the diagram/example above, |
I wanted to chime in here to recommend consideration of Git Flow for the branch/tag/release mechanics. I think it is largely compatible with the broad ideas of SIP-12 described above, but has some benefits:
It also is built around merging and does not include cherry-picking as a matter of course, which keeps history correct and intact. Cherry picks would be rare and used only when absolutely necessary. A nutshell version of what the above SIP-12 process might look like with gitflow:Note, gitflow uses a Daily Development
Releasing
A Note about SemverI just wanted to note that I've had success in the past using a semver scheme where the "version" of a build is automatically generate using a combination of:
For example, imagine the repo's most recent release tag is If I decided this is a minor version bumper (new feature!), I would modify the This sort of version string (which is 100% semver compliant) makes it very easy to go from user-facing version strings to the exact source commit that produced it. This is very useful when trying to reproduce bug fixes or confirm that a deployment actually happened, etc. |
Hi @DiggidyDave, Thank you for your comments. When drafting this proposal, @john-bodley, @michellethomas and I did consider gitflow and in some aspects it is similar with the process above. However, there are a few differences:
Oftentimes we do not want to introduce all new features into the current stable release but rather apply bug fixes to improve its stability and wait for the next minor release to adopt new features. In gitflow, once you are at Please also keep in mind that Superset is a UI project, in which a new feature that does not break the previous version could still greatly affect the user experience. The choice to upgrade from Another key point of this SIP-12 is we also wanted to clarify the details about time frame and set the right expectation for releases, not just the git workflow. These details are organization-specific and not specified in gitflow. We research multiple well-known open-source projects such as Again, I appreciate your thoughts and if there are aspects of gitflow that I misunderstood please feel free to discuss more. References
|
Thanks for reply @kristw ... I understood that SIP-12 was largely about release schedule etc, etc., and I think we agree that it is largely separable from the implementation in terms of branch mechanics (ie not specified and needn't be by something like gitflow, which just provides the abstractions for implementing it). To respond to one point you raised: patching previous releases is very doable in Git Flow. For example, see here: https://stackoverflow.com/a/33052352/3407502 That is what I intended to address in "Daily Development, Item 2" of my comment. Applying the fix on the earliest supported release and merging forward through subsequent supported releases has the additional benefit of completely avoiding cherry-picks and throwing away the history. Avoiding cherry picks is, IMO, the biggest benefit of switching to GitFlow. Otherwise, much of the react process is transferable... cherry-picking seems like an implementation detail in their model, and it looks like it causes them a lot of trouble, see here for example. I'm actually surprised to see react uses cherry picks so heavily, as it throws away history and context and is prone to producing bugs as you lose track of which branches have changes, where they originated, where subsequent edits were made etc, and with a little bit of rigor they are almost completely avoidable. A little discipline around release management (ie directing bug PRs towards the correct fix branch) would allow such patches to be smoothly and unidirectionally merged, with history, in a much less risky, chaotic way, to all of the branches, IMO. Edit: even thought I'm sure y'all have seen these before, I wanted to include a couple of posts from the "please don't cherry pick" viewpoint for posterity :-): |
but if the fix is applied on `support/1.2.x` branch according to this
diagram, the fix will never be merged back to master.
https://gitversion.readthedocs.io/en/latest/git-branching-strategies/gitflow-examples/#support-branches
How could we apply the same fix for both `master` and `support/1.2.x`
without cherry-pick?
*Support Branches*
… Support branches are not really covered in GitFlow, but are essential if
you need to maintain multiple major versions at the same time. You could
use support branches for supporting minor releases as well. If you are just
supporting the majors, then name your branch support/<major>.x (i.e
support/1.x), to support minors use support/<major>.<minor>.x or
support/<major>.<minor>.0. (i.e support/1.3.x or support/1.3.0)
On Fri, Mar 22, 2019 at 11:35 AM Dave Smith ***@***.***> wrote:
Thanks for reply @kristw <https://github.com/kristw> ... I understood
that SIP-12 was largely about release schedule etc, etc., and I think we
agree that it is largely separable from the implementation in terms of
branch mechanics (ie not specified and needn't be by something like
gitflow, which just provides the abstractions for implementing it).
To respond to one point you raised: patching previous releases is very
doable in Git Flow. For example, see here:
https://stackoverflow.com/a/33052352/3407502 That is what I intended to
address in "Daily Development, Item 2" of my comment. Applying the fix on
the earliest supported release and merging forward through subsequent
supported releases has the additional benefit of completely avoiding
cherry-picks and throwing away the history.
Avoiding cherry picks is, IMO, the biggest benefit of switching to
GitFlow. Otherwise, much of the react process is transferable...
cherry-picking seems like an implementation detail in their model, and it
looks like it causes them a lot of trouble, see here
<react-native-community/discussions-and-proposals#53>
for example. I'm actually surprised to see react uses cherry picks so
heavily, as it throws away history and context and is prone to producing
bugs as you lose track of which branches have changes, where they
originated, where subsequent edits were made etc, and with a little bit of
rigor they are *almost completely avoidable*.
A little discipline around release management (ie directing bug PRs
towards the correct fix branch) would allow such patches to be smoothly and
unidirectionally merged, with history, in a much less risky, chaotic way,
to all of the branches, IMO.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#6131 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABlTe1RBSPBaVrfj6iaXrUKQpedgiAEEks5vZSJegaJpZM4Xkuj3>
.
--
*Krist Wongsuphasawat, PhD*
http://kristw.yellowpigz.com
|
Since people like working against master, let's assume gitflow with Before the hotfix we have: Summary of process...
After the hotfix we have: |
I'm going to go ahead and close this due to age, but @john-bodley please feel free to re-open and kick start the |
Motivation
Historically organizations partaking in the release process have managed their own fork of incubator-superset which is deployed in production via a Git submodule. Both companies have been actively developing Superset and deploy internally (features and bug fixes) at a higher cadence than the public PyPI releases. These forks came about because of the potential instability of the master branch (see SIP-13) and provided a mechanism to cherry-pick specific commits thus ensuring an increased stability.
These organizations have discussed ways of trying to fold these forked branches back into the incubator-superset repo to better ensure consistency and to serve as a mechanism for formalizing the release process which is somewhat sporadic.
Proposed Change
The proposed solution is based primarily on having release branches where certain organizations are the primary beta testers for stabilizing the branch.
Master branch
There will be no changes to how the
master
branch currently works. We will do our best to keep themaster
branch healthy in part by the initiatives outlined in SIP-13. If you create a pull request, please do it against themaster
branch. We will maintain stable branches for releases separately but we don’t accept pull requests to them directly. Instead, we will cherry-pick non-breaking changes frommaster
to the two latest releases.Release Cadence
We propose a regular release cadence of every two weeks. This period is a tradeoff between stability and flexibility ensuring that new features can be deployed within a reasonable time frame. Furthermore the longer the release cadence the greater the potential conflict when cherry-picking commits. A release will be deployed two weeks after it was cut.
We propose that each release branch is fully managed by a single organization.
Branch Naming
We propose that all Git branches follow the
<prefix>/<suffix>
naming convention:<username>/<issue>
, i.e.,jane-doe/issue-123
feature/<name>
, i.e.,feature/dashboard-v2
release/<version>
, i.e.,release/1.1
Note if
/
is troublesome we suggest using a two hyphens (--
) instead.Semantic Versioning
Superset follows semantic versioning (
<major>.<minor>.<patch>
) although the major version has remained fixed at zero. We should release patch versions for bug fixes, minor versions for new features, and the major version for breaking changes. When we make breaking changes, we should also introduce deprecation warnings in a minor version so that our users learn about the upcoming changes and migrate their code in advance.Git release branches represent the
<major>.<minor>
level whereas Git tags (associated with a specific commit) represent the fully defined<major>.<minor>.<patch>
version. Note tags may contain an optional release candidate suffix (-rc
).The Git Ecosystem
The following diagram describes the
apache/incubator-superset
Git ecosystem and how we plan to couple this with several organization's internal deployment (superset-internal which uses a Git submodule) which will help serve as a mechanism for stabilizing a release.Day 0
On day 0 the next release branch (
release/1.1
) will be cut frommaster
which references the latest commit. The commit will be tagged withv1.1.0-rc
(the v1.1.0 release candidate) and deployed to PyPI.Day 0 - 13
Only commits associated which resolve bugs present in the release candidate will be cherry-picked onto the release branch. Fixes to features added after the cut will not be cherry picked (†). Note there may be potential merge conflicts (which will have to be resolved) when cherry-picking a fix due to an upstream commit which is not in the release (‡). If this is too complex the cherry-pick should be skipped (---) as it will be included in the next release. New features can be committed on
master
if and only if they’re defined in their entirety within the two week cycle. Larger features should be developed on a feature specific branch (which is then merged intomaster
) or using feature flags.The following diagram shows the use of a feature branch which spans multiple releases.
Day 7
Organizations partaking in the release process will update the Git submodule reference to point to the release (
release/1.1
). This seven day lag provides a soak period of ensuring that critical bugs are not deployed to our production environments. The update is made to the.gitmodules
file,which ensures it tracks the correct release branch. Like the two week release process the submodule branch will be updated every two weeks.
Day 7 - 20
Periodically organizations partaking in the release process will ensure that the submodule in their internal deployment will reference the latest commit from the remote branch (which contains all the cherries) via,
which will fetch the latest changes for each submodule, merge them in, and check out the latest revision of the submodule. This step allows us to further test and stabilize the branch in a production environment.
Day 14
The first (non release candidate) version will be tagged (
v1.1.0
) and deployed to PyPI which references the latest commit on the branch (*). Simultaneously the next release branch (release/1.2
) will be cut frommaster
which will mimicv1.1.0
except it will also include any feature commits which were merged in the past 14 days. Note a new release branch is cut if and only if feature commits were added, to prevent duplication.The following diagram shows an example where on day 14 a new release branch is not cut as there were no new features added since day 0. The next viable release cut date will then be day 28 to preserve the release cadence.
Day 14 - 27
Additional bug fixes to commits in the release candidate will be cherry-picked onto the n most recent releases (where n = 2, i.e.,
release/1.1
andrelease/1.2
). Throughout this period the tag can be updated (incrementing the patch) as desired based on the severity of the fix.Day 28 -
Assuming there is a new release branch (
release/1.3
) we will no longer be picking cherries ontorelease/1.1
since we only support two releases (release/1.2
andrelease/1.3
). In summary most release branches will have a four week active lifespan.Branches should exist for infinitum to ensure critical security fixes can be patched to the necessary branch.
Database Migrations
Database migrations (which are handled via Alembic) have an additional lineage component (used to determine upgrades and downgrades) which further complicates matters. We need to ensure the playback of versions is correct when upgrading to a newer release and that a downgrade is never necessary.
The logic for cherry-picking migrations is similar to above whereas migrations can be cherry-picked onto a release branch if and only if all of its ancestors are present on the branch. Given that there no notion of resolving merge conflicts in Alembic, any migration (and thus future migrations) related to a feature will be skipped (
--
) and included in the next release.Action Items
Currently we plan on manually cherry-picking the commits from
master
to the desired release branch. This could be automated in the future based on PR tags and/or the assistance from a bot.We also should check with Apache whether there are any concerns/feasibility issues with releasing on a two week cadence.
@beto @mistercrunch
The text was updated successfully, but these errors were encountered: