The Foreman Release Engineering scripts

A collection of small scripts, adhering to the Unix philosophy, used to release The Foreman. They automate parts of the branch and release processes. It is also used for the Katello branch and release processes.

The most important environment variables are PROJECT and VERSION.

Procedures

Procedures are kept in the procedures directory.

Roles

Both the branch and release procedures defines two roles: an owner and an engineer. Someone can be both owner and engineer at the same time, and it is preferable for a fast process. Having both in (very) different timezones can lengthen the process. Sometimes it's just not possible and multiple people perform the procedures. Below is a listing of access used.

Foreman release owner

• Belong to the Foreman GitHub release-managers team
• Belong to the releases group on Discourse
• Matrix: permission to change the topic in #theforeman on matrix.org. See Matrix moderation on how channel admins can grant this.
• Redmine access: be added to the Developers group
• Transifex account (unclear on the exact permissions to extract translations)

Foreman release engineer

• Belong to the Foreman GitHub release-engineering team
• Foreman infrastructure access including secrets
• Jenkins access
• stagingyum access
• theforeman Fedora group member which can be viewed through Fedora Account System

Katello release owner

• Write permission to Katello/katello
• Push access to the Katello gem
• Belong to the Foreman GitHub katello-release-owners team
• Belong to the releases group on Discourse
• Redmine access: be added to the Developers group

Katello release engineer

These are the same as Foreman's release engineer.

Branching

Use procedure_branch to display the steps. Modify PROJECT and VERSION as needed.

PROJECT=foreman VERSION=3.7 ./procedure_branch

It will have created releases/${PROJECT}/${VERSION}/settings, which should be submitted as a pull request.

A more advanced invocation is to specify the parameters directly and copy the output. This uses wl-copy for Wayland; xclip achieves the same for Xorg.

PROJECT=foreman VERSION=3.7 ./procedure_branch 2023-05-23 TheOwner TheEngineer | wl-copy

Now post this to Discourse Development/Releases as $PROJECT $VERSION branching process. Follow the process as instructed.

Release

First make sure you have fetched, pulled in the latest commits and rebased your fork. Secondly ensure releases/${PROJECT}/${VERSION}/settings contains the correct FULLVERSION. Modify as needed and submit as a pull request. Then generate the procedure, similar to the branching process:

PROJECT=foreman VERSION=3.7 ./procedure_release

Or the complete version:

PROJECT=foreman VERSION=3.7 ./procedure_release 2023-05-23 TheOwner TheEngineer | wl-copy

Now post this to Discourse Development/Releases as $PROJECT $FULLVERSION release process. Follow the process as instructed.

Setup

Python and python-jenkins are needed. On Fedora this can be installed:

dnf install python3-jenkins

Foreman requires GPG signed git tags. Configure git with your personal gpg key id as your signing key:

git config --global user.signingKey <YOUR_PUB_KEY_HASH>

Gopass

For storing secrets, gopass is used. On Fedora it can be installed:

dnf install gopass

Then make sure you have a GPG key on your system and initialize gopass with your key:

gopass init <YOUR-PUB-KEY-HASH>

For running jobs in Jenkins, make sure you have access and add your Jenkins password or API token:

gopass edit theforeman/jenkins-token --create

Copr

Copr is where all packages are built and the stage repositories are generated from for releases. API access is needed in order to perform release activities. To ensure you have Copr access setup:

1. Go to https://copr.fedorainfracloud.org/api/
2. Login if you are not already
3. Follow the directions

Release engineers

For commands on the Foreman infrastructure, add your personal sudo password:

gopass edit theforeman/unix --create

The releases store from the shared secret storage is also needed:

gopass clone secrets.theforeman.org:/srv/secretsgit/theforeman-release.git theforeman/releases

Importing an existing release

When a GPG key has already been generated, it can be imported from the backups:

import_gpg_private

Typical release flow

Make sure VERSION is correct in settings and FULLVERSION in releases/$PROJECT/$VERSION/settings. This assumes a GPG key is already present.

./tag_project
./release_tarballs
./download_tarballs
./inspect_tarballs
./sign_tarballs
./bump_deb_packaging
./bump_rpm_packaging
./release_packages
# These steps can happen during the build after RPMs have been built but DEBs are still running
./generate_stage_repository
./sign_stage_rpms
./upload_stage_rpms

When handling non-Foreman releases (currently supported: Katello and Client), set PROJECT to the lowercase name of the project and VERSION to the version of the project (if it differs from the Foreman one).

PROJECT=client ./generate_stage_repository
PROJECT=katello VERSION=3.13 ./generate_stage_repository

Generating a new GPG Key for a X.Y release

When starting a new release, the following scripts can be used to generate a new key:

generate_gpg
export_gpg_private
export_gpg_public
sign_gpg
upload_gpg

Generating a new GPG Key for signing the Debian repository

Our Debian repository doesn't rotate keys based on releases, but on a time basis.

To generate a new key:

export PROJECT=foreman-debian
export VERSION="$(date '+%Y')"
generate_gpg
export_gpg_private
export_gpg_public
sign_gpg
upload_gpg

Settings and customizing them

Settings can be customized in settings.local. The following settings are supported:

GIT_DIR=$HOME/dev # Projects are cloned here
GIT_REMOTE=upstream # Git remote for cloned projects
PACKAGING_PR=true # Create a PR in bump_{deb,rpm}_packaging

Generating Stage Repository

The build_stage_repository script generates a stage repository locally that can then be uploaded to the staging repository server or used locally for testing. This is done for RPMs and SRPMs. The workflow for each is:

For releases (executed by release owner):

1. Uses reposync to copy all RPMs from production (yum.theforeman.org) to a local repository
2. Run repodiff comparing the local copy of production and Copr for the release
3. Download the new packages from Copr
4. Filter all packages through comps file, removing anything not in the comps file
5. Runs createrepo
6. Generates a new module metadata file based upon the local repository
7. Generate list of unsigned RPMs
8. Sign the unsigned RPMs
9. Update the repository metadata

For nightly (executed by Jenkins):

1. Copy all RPMs from Copr for the given repository to local
2. Filter the downloaded RPMs through comps file, removing anything not in the comps file
3. Runs createrepo
4. Generates a new module metadata file based upon the local repository

The local repository can then be uploaded to the server backing stagingyum.theforeman.org using rsync. Where any RPM found locally already in staging is kept, and any RPM found in staging not in the local repository is removed.

Performing the release workflow

The release workflow is execute using the following actions:

generate_stage_repository
sign_stage_repository
upload_stage_repository