Proposal: Dry-run release before each full release

[adamfarley]

As discussed in the October retrospective, this issue debates the pros, cons, and specifics of the following changes to the release process.

1. That several repos (build script, test maybe, full list tbd) be "frozen" a week (or more) prior to the release.
2. That the community conducts a "dry-run", which is essentially the full release process sans publish.
3. That the results of the dry-run be reviewed, and any issues addressed as a matter or urgency, prior to the full release.
4. That the repos be unfrozen after the conclusion of the release.

Once we have a consensus, if the community elects to go through with the change (or one like it), we will need to:

a) Update RELEASING.md to cover the "how" for each change.
b) Brief the community (call, slack message, blog post, a combination, etc) to inform them of the changes.
c) Modify the pipelines as needed to enable a dry-run that has all the features of a full release, sans publish.
d) Do a full dry-run, end-to-end, to test the entire process.

The floor is now open for discussion.

[adamfarley]

Also, I remain convinced of the merits of adding a step 5: To store a copy of the build scripts in a separate branch when the build script repos are unfrozen. This, or a change to enable us to run against a specific build script commit level.

The reasoning for this is that is if we get a re-release, we have a stable copy of the build scripts we can use. Sure, we could leave the build script repos frozen for a while after the release, but for how long? And what do we do if the re-release is declared the day after we've unfrozen the repos and merged everyone's pet project?

[aahlenst]

Also, I remain convinced of the merits of adding a step 5: To store a copy of the build scripts in a separate branch when the build script repos are unfrozen. This, or a change to enable us to run against a specific build script commit level.

Agree, but I prefer to flip it: Tag/branch before release.

Not having branches/tags is a problem for reproducibility. On top of that, we get re-releases that do not have the same feature set as the original. For example, 15.0.1+9.1 for macOS contains already the new trust store slated for release with the January updates.

Locking down the repositories has further drawbacks:

• People have accidentally merged changes in the past during lockdown.
• If releases take a lot of time (as has happened multiple times in the past), we end up with a very large queue of PRs.

[adamfarley]

Also, I remain convinced of the merits of adding a step 5: To store a copy of the build scripts in a separate branch when the build script repos are unfrozen. This, or a change to enable us to run against a specific build script commit level.

Agree, but I prefer to flip it: Tag/branch before release.

We should ensure the build scripts are stable prior to branching/tagging, so perhaps we should do it after the dry-run has confirmed things are ok, but before the full release?

Locking down the repositories has further drawbacks:

* People have accidentally merged changes in the past during lockdown.

* If releases take a lot of time (as has happened multiple times in the past), we end up with a very large queue of PRs.

I raised these concerns at the retrospective (great minds), however I think the response to the former was that there's a special kind of lockdown that we didn't use previously, and that the use of this would be a more effective block against merges. I don't remember the details though, so perhaps someone else can speak more authoritatively on that.

As for the latter drawback, I remember wanting to mention this, but I don't recall if I did so.

In short, I absolutely agree that a "release" branch is the right way forwards. A tag in the master branch can be a good idea, but if we don't freeze the master branch, we'd lack the ability to add critical build script changes to later builds without cherry-picking. A separate branch allows us to double-commit key changes during the release (where needed), while keeping non-vital commits in the master branch.

So, perhaps a release branch with tags?

[aahlenst]

We should ensure the build scripts are stable prior to branching/tagging, so perhaps we should do it after the dry-run has confirmed things are ok, but before the full release?

I find it odd that we need a dry-run despite having nightly builds. So if nightly builds aren't sufficient now, we should work towards getting there.

So, perhaps a release branch with tags?

Yes. We need both. Releases should happen from tags. And we can have a per-release branch (something like release-20.10 for October 2020 patch update) that we update as necessary while putting the same fixes into master, too.

[andrew-m-leonard]

I find it odd that we need a dry-run despite having nightly builds. So if nightly builds aren't sufficient now, we should work towards getting there.

I suspect the "Release" build option does have a few differences compared to "Nightly", but not many I would hope, but I agree assuming a "lockdown" is in place for PR changes, then Nightlies should be satisfactory.
I question whether we should make nightlies simply "Release with no Publish"?

[smlambert]

Release builds run more testing. Related: https://github.com/AdoptOpenJDK/openjdk-build/issues/2215

The more I think about it, the more I think that we could change the nightly schedule to run on the 5 weekdays,
then have a weekly schedule that runs a release build with no publish at the start of the weekend, giving it the entire weekend to run through the additional tests (and accomplishing the goal originally set out by openjdk-build/2215 of getting those extra tests running on a regular basis).

[smlambert]

But yes, I agree that if there are other differences between nightly and release builds (besides the amount of testing they launch), those differences should be removed

[karianna]

We do this at Adoptium now.