Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs:/streams: try to explain streams lifecycle #1736

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Docs:/streams: try to explain streams lifecycle #1736

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 34 additions & 12 deletions stream-tooling.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Introduction

FCOS will have multiple streams:
FCOS have multiple streams:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
FCOS have multiple streams:
FCOS has multiple streams:


| Type | Name (SCM branch and ostree ref) | ostree repo |
| -- | -- | -- |
Expand All @@ -13,8 +13,6 @@ FCOS will have multiple streams:
| Development | next-devel | annex |
| Mechanical | rawhide | annex |
| Mechanical | branched | annex |
| Mechanical | bodhi-updates | annex |
| Mechanical | bodhi-updates-testing | annex |

Development and mechanical streams are subject to change.

Expand All @@ -27,20 +25,18 @@ We need a way to both (1) fix the content set for a particular stream release, a
- Koji tags: a way to track packages built in Koji. Koji is capable of creating yum repos from such tags. RPM builds may be "tagged" in so that the next repo regeneration includes it.
- [dist-git](http://src.fedoraproject.org/): git where RPM spec files are kept and Koji builds source from.

## Proposal
## Existing streams

**Mechanical** streams are not curated; they're automated nightly snapshots of the underlying repos. They source their RPMs from the regular Fedora repos (using 30 here to mean `$currentrelease`):
1. **rawhide** <- f32
2. **branched** <- f31 when a branch exists, otherwise tracks **rawhide**
3. **bodhi-updates** <- f30-stable + f30-updates
4. **bodhi-updates-testing** <- f30-stable + f30-updates + f30-updates-testing
**Mechanical** streams are not curated; they're automated nightly snapshots of the underlying repos. They source their RPMs from the regular Fedora repos:
1. **rawhide** <- fedora rawhide repos
2. **branched** <- fedora-branched when a branch exists, otherwise tracks **rawhide**

**Production** streams are intended for production use. They source their RPMs from a _single_ Koji tag, `coreos-pool`, from which we create a yum repo:
1. **next** <- coreos-pool
2. **testing** <- coreos-pool
3. **stable** <- coreos-pool

**Development** streams are nightly snapshots of content headed for the production streams. There's one development stream at the base of each promotion path; thus, **stable** doesn't have one because it promotes from **testing** instead. **next-devel** will only be maintained in the periods where **next** is independent of **testing**. Development streams source their RPMs from a _single_ Koji tag, `coreos-pool`, from which we create a yum repo:
**Development** streams are nightly snapshots of content headed for the production streams. There's one development stream at the base of each promotion path; thus, **stable** doesn't have one because it promotes from **testing** instead. **next-devel** will only be maintained in the periods where **next** is independent of **testing** (when a fedora beta is released, **next-devel** moves to the beta content). Development streams source their RPMs from a _single_ Koji tag, `coreos-pool`, from which we create a yum repo:
c4rt0 marked this conversation as resolved.
Show resolved Hide resolved
1. **next-devel** <- coreos-pool
2. **testing-devel** <- coreos-pool

Expand All @@ -52,10 +48,36 @@ There is also a second Koji tag, `coreos-release`, for packages which have been

We maintain a git repository containing the rpm-ostree treefile and lockfiles. This could be [fedora-coreos-config](https://github.com/coreos/fedora-coreos-config). We have one branch for each stream, and no main branch.

For the mechanical streams, a nightly job will run the compose from the corresponding yum repos and SCM refs. This job will output a lockfile for each CPU architecture. Those lockfiles will be committed to Git to preserve a record of the build's contents, and the builds will be pushed to the corresponding ostree refs. The {bodhi-updates, branched} lockfile will also be PR'd to the {testing-devel, next-devel} branch, the latter only during the part of the cycle where next-devel is maintained. We want to keep the development branches ready to release, so those PRs are not merged unless green.
For the mechanical streams, a nightly job will run the compose from the corresponding yum repos and SCM refs. This job will output a lockfile for each CPU architecture. Those lockfiles will be committed to Git to preserve a record of the build's contents, and the builds will be pushed to the corresponding ostree refs. The branched lockfile will also be PR'd to the {testing-devel, next-devel} branch, the latter only during the part of the cycle where next-devel is maintained. We want to keep the development branches ready to release, so those PRs are not merged unless green.

The lockfiles produced from the automatic snapshot will never be hand-modified, and in the next/testing/stable branches will never be modified at all except during promotions. Instead, pins (to older NEVRAs) and updates (to newer ones) will be hand-maintained in the Git branches in a separate lockfile that overrides the autogenerated ones. These overrides will be the major distinction between the mechanical refs and the "curated" (development/production) refs. Each curated branch will have one override file, which can carry both CPU-architecture-independent and architecture-specific overrides.

### Streams Lifecycle

#### Regular State
Although all the streams are built from scratch (there is no tangible relations between streams) here is how the RPMs flow towards stable:

The **testing-devel** Stream is built daily with the content of the `fedora-update` repository. If the build successfully pass the tests,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which fedora-update repository are you referring to here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://docs.fedoraproject.org/en-US/quick-docs/fedora-repositories/#the-updates-repository

the resulting lockfiles are committed to the **testing-devel** branch of `fedora-coreos-config`.
Any new RPM receive the `coreos-pool` koji tag.

For a release (approximately every two weeks), someone manually triggers the `bump-lockfile` job to propagate the **testing** content to the **stable** branch.
The **testing-devel** content is propagated to the **testing** and **next** streams. Theses images are built from the `coreos-pool` yum repository.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it not testing-devel -> next -> testing -> stable? Reading this I feel you're saying that testing-devel goes directly to both ( testing <- testing-devel -> next).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when Fedora is not branched (right now), testing and next are the same .

Looking at testing commit history : it's at testing devel e13ff7d

Looking at next commit history : it's at testing devel e13ff7d


#### When Fedora Branched exist

Before every released, there is a stabilisation period. A branch is created from rawhide, the next release enters it's [branched](https://docs.fedoraproject.org/en-US/releases/branched/) state.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit-pick: Before every release, ...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Before every released, there is a stabilisation period. A branch is created from rawhide, the next release enters it's [branched](https://docs.fedoraproject.org/en-US/releases/branched/) state.
Before every release, there is a stabilisation period. A branch is created from rawhide, the next release enters its [branched](https://docs.fedoraproject.org/en-US/releases/branched/) state.

During that time, the **next-devel** stream is enabled.
The **branched** stream is built daily with the content of the `fedora-branched` repository. If the build successfully pass the tests,
the resulting lockfiles are committed to the **next-devel** branch of `fedora-coreos-config`.
**next-devel** is then built daily.
When the branched release of fedora is promoted to `beta`, the **next-devel** content is promoted to **next** during the release process, as described above.

#### Rawhide

The **rawhide** stream is daily built from the `fedora rawhide packages. When there is no `fedora-branched` release, the **branched** stream mirrors
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit-pick: missing backtick.

**rawhide**. This stream allow us to monitor changes early and react accordingly. The results are not used in any lockfiles.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's interesting... honestly, this is the first time I'm considering the lifecycle as a whole. I feel like a chart would help clarify things described above ;)


### How will releases happen?

When it's time to cut e.g. a promoted testing release, we push a merge commit to the testing branch which takes the testing-devel branch's tree in entirety. (That is, we use the Git "theirs" merge strategy, which, helpfully, doesn't exist.) We then tag the Git commit on the testing branch and do an official build and CI run. In the absence of flakes (ha!) this will pass because the commit is known to be green. If the build is bad, we abandon the release and iterate. Otherwise, we proceed with releasing artifacts.
Expand All @@ -74,7 +96,7 @@ Update the development treefile as usual. On the next bot push, the lockfile wil

To focus development effort, there will be one base treefile shared across all branches, whose canonical copy will live in the testing-devel branch. Changes will automatically be mirrored to next-devel and to the mechanical branches. To address divergence across Fedora releases, each branch will also have an overlay treefile (possibly empty):

- **testing-devel** -> automatically mirrored to bodhi-updates and bodhi-updates-testing
- **testing-devel**
- **next-devel** -> automatically mirrored to branched
- **rawhide**

Expand Down
Loading