Release Flatcar Container Linux Alpha 3874.0.0, Beta 3850.1.0, Stable 3815.2.0, LTS 3510.3.2

[sayanchowdhury]

The release of the following Flatcar Container Linux version is planning February 6th , 2024.

• Alpha 3874.0.0
• Beta 3850.1.0
• Stable 3815.2.0
• LTS 3510.3.2

1. Preparation

Owner: @sayanchowdhury

• A. Go/No-Go hackmd doc created from HackMD template for test tracking and release notes: (https://hackmd.io/JUCZzDkNSwm2S49FlgtW3A)
• B. All planned changes are merged & blocker bugs are fixed, last nightly reviewed for problems.
• C. Check that the kernel PR are merged for the channel(s) to be released.

2. Build

Owner: @sayanchowdhury

• A. Branched off the main branch (only for new major).
• B. Tagged all the build branches.
• C. Started the Jenkins builds, post the links into this issue as comment.

3. Release

i. The Go/No-Go Meeting

Owner: @sayanchowdhury

• A. Test status captured in the HackMD document.
• B. All tests are Green.
• C. Release notes / announcements added to Go/No Go doc from container/image_changes job by selecting Timestamps: None.
• D. QA: Checked the diff for image file list, packages, and image size in the container/image_changes job output.
• E. Conduct Matrix Go/No Go meeting (inc. release notes review).

ii. Artifacts goes public

Owner: @sayanchowdhury

• A. Created the Release Notes on the scripts repo for the respective tags on GitHub as GitHub Releases.
• B. Ran the container/release Release job.
• C. Images uploaded with copy-to-origin.sh.
• D. Symlink to "current" updated with set-symlink.sh.
• E. Website updated with ./update-flatcar-versions.sh and PR merged.

iii. Payload

Owner: @pothos

• A. Update payload signed/uploaded.
• B. Release package published in Nebraska (Prerequisite: 3.ii.E should marked done).

iv. Cloud Marketplace & Cluster API

Owner:

• A. Azure (@sayanchowdhury)

  • Offers updated and started publishing.
  • Sent the offers to publish to Live.

• B. GCP (@t-lo)

  • Offer deployment package uploaded to Google Bucket & submit for verification.
  • Offer Publish

• C. Cluster API for new Stable (@sayanchowdhury)

  • Azure: Manually run ./azure-sig.sh to create New Shared Community Gallery Images
  • OpenStack: documentation

4. Announcements

Owner: @tormath1

• A. Sent announcement to Mailing Lists
• B. Sent Tweet out to Twitter, and Toot out to Mastodon
• C. Brief version announcement in slack (k8s slack #flatcar) and the Flatcar Matrix channel

[sayanchowdhury]

Alpha: http://jenkins.infra.kinvolk.io:8080/job/container/job/sdk/1315/cldsv/ (restarted 2024-02-12)
Beta: http://jenkins.infra.kinvolk.io:8080/job/container/job/packages_all_arches/3438/cldsv/ (restarted 2024-02-12)
Stable: http://jenkins.infra.kinvolk.io:8080/job/container/job/packages_all_arches/3415/cldsv/ (restarted 2024-02-12)
LTS: http://jenkins.infra.kinvolk.io:8080/job/container/job/packages_all_arches/3437/cldsv/ (restarted 2024-02-12)

[pothos]

We should also build LTS, I'll do it now

[sayanchowdhury]

Update: The team is working on the last few flaky tests, once green we will proceed with the release.

[t-lo]

Started the GCP publishing process; offer is currently under review. Will publish as soon as its ready.

[ewassef]

This version is broken. Containerd is not coming up

[t-lo]

This version is broken. Containerd is not coming up

@ewassef Which channel, and on what cloud / infrastructure (vendor)?

[t-lo]

GCP marketplace offer is live.

[ewassef]

Aws stable. Had to roll back to 3760

[t-lo]

@ewassef I'm very sorry for the bad experience!

I just checked our AWS release tests for 3815.2.0 - we did not observe any indication of containerd malfunctioning (docker and kubernetes tests passed).

Could you please open a bug and provide diagnostics (journald output etc.) so we can investigate?

[t-lo]

If you prefer to chat with maintainers directly to interactively debug this, we're over at Matrix: https://app.element.io/?updated=1.11.53#/room/#flatcar:matrix.org

[ewassef]

OK I'll get some colleagues to join here and provide details

[t-lo]

OK I'll get some colleagues to join here and provide details

@ewassef did #1363 help to address the issue?

[ewassef]

Yes, that seems to be the problem. Thanks for the quick turnaround identifying the issue

[t-lo]

We're here for that! Thanks for reporting it, this will help other users who might run into the issue.

[sayanchowdhury]

There are some issues with the Azure LTS publishing which we are looking into. The release will be coming out next week.

[tormath1]

Yes, that seems to be the problem. Thanks for the quick turnaround identifying the issue

@ewassef happy that the issue is solved! For your information, this change has been in Beta since Jan 18, 2024 - we consider Beta robust enough to deploy a few Beta nodes in your workloads. This is a great and simple way to contribute to Flatcar: helping team and users by identifying any issues before they land on Stable. It's valuable for both side.

[ewassef]

Thanks! Do yall publish the Amis on the public cloud in beta?

[t-lo]

Yes; all cloud offers should have Alpha, Beta, and Stable channels.

[tormath1]

@ewassef here's the AMIs for beta: https://beta.release.flatcar-linux.net/amd64-usr/current/flatcar_production_ami_all.json

[TimoKramer]

There are some issues with the Azure LTS publishing which we are looking into. The release will be coming out next week.

Stable release is also affected. 3815.2.0 is not available on Azure, 3760.2.0 is available.

[sayanchowdhury]

There are some issues with the Azure LTS publishing which we are looking into. The release will be coming out next week.

Stable release is also affected. 3815.2.0 is not available on Azure, 3760.2.0 is available.

One of the offers is still in publishing state, and should be resolved by EOD. The LTS offer will take a few more working days to resolve

[TimoKramer]

Still the latest stable image is not available...

[sayanchowdhury]

The Azure images are updated to the latest Stable now.