Jenkins Release

Table of Contents

• Introduction
• Settings
• Process
  • Introduction
  • Release
  • Packaging
• Profile
  • Weekly
  • Stable Release Candidate
  • Stable
  • Security
• Certificate
• Core Maintainers
• FAQ
• Miscellaneous

Introduction

This repository contains everything needed to release Jenkins core.

This includes:

• Documentation how to release Jenkins LTS releases, available at doc/releases.md.

• Jenkinsfiles for Releasing and Packaging.

• Kubernetes POD templates definition for releasing and packaging environments on Linux and Windows operating systems.

• Any configuration or utility scripts

The release process implies following Git repository:

• jenkinsci/jenkins or any other git repository containing the Jenkins code to be released.

• jenkinsci/packaging contains tools and configuration to package Jenkins for OS distribution [Debian, Red Hat, SUSE, MSI, WAR]

• jenkins-infra/chart contains the Helm chart used to deploy release.ci.jenkins.io on top of Kubernetes.

• jenkins-infra/azure contains the terraform code used to provision infrastructure needed by the release environment

Settings

Variable	Description	Release	Packaging
GIT_EMAIL	Defines git email used in git commits.	x	x
GIT_NAME	Defines git name used in git commits.	x	x
GIT_SSH_COMMAND	Defines options that can be passed to every git commands used in release scripts.	x	x
GPG_KEYNAME	Define the GPG key name that will be used to sign artifacts.	x	x
GPG_FILE	Define the GPG file name that will be used to import gpg key into keyring.	x	x
GPG_VAULT_NAME	Define the Azure key vault containing the gpg key.	x	x
JENKINS_ASC	Defines the Jenkins signature file name.		x
JENKINS_DOWNLOAD_URL	Define the url used to download jenkins.war.		x
JENKINS_VERSION	Define which version of Jenkins will be downloaded for packaging. It accepts following values: latest, fetches the latest version available in a maven repository based on the maven-metadata.xml. version, search the latest version available based on version. More information in getJenkinsVersion.py Important Versions available depends on the maven repository defined by MAVEN_REPOSITORY_NAME JENKINS_VERSION is totally independent of RELEASE_PROFILE explained below		x
JENKINS_WAR	Defines the jenkins.war file name downloaded from a maven repository using getJenkinsVersion.py.		x
RELEASE_GIT_REPOSITORY	Defines the Jenkins git repository read by the build. It is also the repository where generated commits and new tags will be pushed.	x	x
RELEASE_GIT_BRANCH	Defines the Jenkins branch used to checkout and then pushes commits related to the release.	x	x
RELEASE_GIT_PRODUCTION_BRANCH	Defines the branch used as the production branch during promotion.		x
RELEASE_GIT_PRODUCTION_REPOSITORY	Defines the git repository used as the production git repository during promotion.		x
RELEASE_GIT_STAGING_BRANCH	Defines the branch used as the staging git branch used to promote to the production one.		x
RELEASE_GIT_STAGING_REPOSITORY	Defines the git repository used as the staging git repository during promotion.		x
RELEASE_GIT_STAGING_REPOSITORY_PATH	Defines path where staging git repository will be be checkout.		x
MAVEN_REPOSITORY_URL	Defines the maven repository URL where is pushed generated artifacts.	x	x
MAVEN_REPOSITORY_NAME	Defines the maven repository name where is pushed generated artifacts.	x	x
MAVEN_REPOSITORY_PASSWORD	Defines the maven repository password.	x	x
MAVEN_REPOSITORY_USERNAME	Defines the maven repository username.	x	x
MAVEN_REPOSITORY_PRODCUTION_NAME	Defines the production maven repository name used during maven promotion.		x
PKGSERVER	Defines where the different packages will be published.		x
PKGSERVER_SSH_OPTS	Defines custom ssh options used to connect to PKGSERVER.		x
PROMOTE_STAGING_MAVEN_ARTIFACTS_ARGS	Defines parameters used by promoteMavenArtifacts.py. Default value is set to item --mode copy --source $MAVEN_REPOSITORY_NAME --destination $MAVEN_REPOSITORY_PRODUCTION_NAME --url $MAVEN_REPOSITORY_URL --username $MAVEN_REPOSITORY_USERNAME --password $MAVEN_REPOSITORY_PASSWORD --search '/org/jenkins-ci/main' $(./utils/getJenkinsVersion.py --version)}"		x
RELEASELINE	Define the release line used by packaging scripts in jenkinsci/packaging.		x
RELEASE_PROFILE	Define a file containing environment variables specific to a release, located in the profile.d directory.	x	x
SIGN_ALIAS	Define code signing certificate name.	x	x
SIGN_KEYSTORE_FILENAME	Define code signing certificate file name.	x	x
SIGN_KEYSTORE	Define signing keystore.	x	x
SIGN_CERTIFICATE	Define code signing certificate file name.	x	x

Process

Introduction

The release process is divided in two categories. The first part that we mention by using the term the release, is when we create a new java code release. It relies on the Maven Release Plugin to perform the release. It involves signing with a GPG key and a code signing certificate. At the end of this operation signed maven artifacts are pushed to a Maven repository.

The second part that we name packaging is when we retrieve from a Maven repository, the version we want to package. Then we build distribution packages, publish them, promote artifacts between staging and production environment if needed and finally we ensure that our mirrors are up to date.

Required

In order to trigger a new release, you must fulfil following requirements:

• ✓ Jenkins private VPN Access access

• ✓ Being part of LDAP group 'release-core'

• ✓ release.ci.jenkins.io access

Release

At this stage, we are going to retrieve the Java code, release a new version using the maven release plugin and then publish artifacts on a maven repository.

It’s important to notice that we do not use the maven release plugin to checkout git repositories neither to push changes. This allow us to release from a different git repository than the one defined in the pom.xml. We also need to be able to push commits to a different repository than the one defined in the pom.xml.

1. localCheckout must be set to true

2. pushChanges must be set to false

Steps

Estimated time +- 1h30

1. Connect to the Jenkins private VPN (private.vpn.jenkins.io).

2. Open your favorite browser to https://release.ci.jenkins.io.

3. Trigger the release job on the master branch. Link.

4. Once triggered, it asks you which release line you want to do. It’s important to know that the release line matches one of the profiles file defined here, so please carefully review the settings and be sure that it does what you are looking for.

5. At the end of the job, git commits and maven artifacts will be pushed to their respective locations.

Validate

To validate that the release went well, excepted by having a green build, you can double-check that your artifacts have been correctly pushed to Maven repository located on $MAVEN_REPOSITORY_URL/$MAVEN_REPOSITORY_NAME/org/jenkins-ci/main/jenkins-war/. You could also run jarsigner -verify <your generated artifact> is correct.

Stage

In order to have private maven releases, you can modify RELEASE_GIT_REPOSITORY or MAVEN_REPOSITORY_NAME, to respectively use code from a private git repository and then push artifacts to a private maven repository.

Artifact promotion is done in the next stage 'packaging'.

Packaging

The packaging process looks after the latest Jenkins version published on a Maven repository and then build and publish artifacts for Debian, Red Hat, SUSE, Windows. It also republishes the War file on the package server. If enabled, then it promotes git commits between git repository, promotes maven artifacts between maven repository.

Note	Packages are not re-published if they already exist, only package website is overridden so it’s safe to re-trigger the job.

Steps

Estimated time +- 30min

1. Connect to the Jenkins private VPN (private.vpn.jenkins.io)

2. Open your favorite browser to release.ci.jenkins.io

3. Trigger the packaging job on the master branch. Link

4. Once triggered, it asks you which release line you want to package for. The release line matches one of the profile defines in profile.d, so please carefully review those settings in order to validate that’s what you are looking for.

Once the job is done, every package will be published and then mirror synchronized.

Validate

Ensure that packages are correctly published on pkg.jenkins.io and correctly signed.

Stage

Staging packages is not yet fully supported, more information on INFRA-1363 and INFRA-2608

Profile

The release profile is used to identify the kind of release we are going to do. They are mainly influenced by following elements:

1. Do we want to releases based on different repository branch?

2. Do we want to release based on different git repository?

At the moment we identify three release types:

1. Weekly

2. Stable

3. Security (both weekly and LTS)

Note	release.ci.jenkins.io has two generic jobs, one for release and a second one for packaging. One job per release type triggers the two generic jobs with different parameters. While it isn’t required to trigger a release type job, it increases visibility and reduces the risk of human error.

Weekly

The weekly release is the default release. It is scheduled every Tuesday as defined by this cron. It uses parameters defined in this file

It releases using the repository jenkinsci/jenkins from the branch master. Artifacts are pushed to the default maven repository 'Releases'.

If for some reason the release job needs to be re-triggered, you can:

1. Connect to the Jenkins private VPN (private.vpn.jenkins.io)

2. Open your favorite browser to release.ci.jenkins.io

3. Review the weekly environment file:

4. Trigger the weekly job

Note	You can re-trigger individually the two downstream jobs, release and packaging. Re-triggering the release will do a version bump then push new artifacts. Re-triggering the packaging job won’t published artifacts if they already exist but it will update website html.

Stable Release Candidate

A stable release-candidate is a manually triggered release that happens around once a month. It uses parameters defined in this file.

Before triggering a new stable release candidatae release, some steps are required:

1. Prepare jenkinsci/jenkins repository → missing documentation link.

2. Create a branch on jenkins-infra/release with a branch name that match the release branch from jenkinsci/jenkins like rc-stable-<jenkins_version>.

3. Review and update the stable environment file with:

   1. RELEASE_GIT_BRANCH set to the jenkinsci/jenkins release branch like stable-2.235

   2. PACKAGING_GIT_BRANCH set to the appropriate jenkinsci/packaging branch, e.g. stable-2.235

4. Trigger the stable job

Note	You can re-trigger individually the two downstream jobs, release and packaging. Re-triggering the release will do a version bump then push new artifacts. Re-triggering the packaging job won’t published artifacts if they already exist but it will update website html.

Stable

A stable release is a manually triggered release that happens around once a month. Refer to LTS Release Line for more detailed information. It uses parameters defined in this file.

Before triggering a new stable release, some steps are required:

1. Prepare jenkinsci/jenkins repository → missing documentation link.

2. Create a branch on jenkins-infra/release with a branch name that match the release branch from jenkinsci/jenkins like stable-<jenkins_version>.

3. Review and update the stable environment file with:

   1. RELEASE_GIT_BRANCH set to the jenkinsci/jenkins release branch like stable-2.235

   2. JENKINS_VERSION set to the final release version that will be packaged. If set to 'stable' then the packaging job will try to guess the version based on what was pushed to the maven repository. cfr settings.

   3. PACKAGING_GIT_BRANCH set to the appropriated jenkinsci/packaging branch

4. Trigger the stable job

Note	You can re-trigger individually the two downstream jobs, release and packaging. Re-triggering the release will do a version bump then push new artifacts. Re-triggering the packaging job won’t published artifacts if they already exist but it will update website html.

Security

The security release follows the same process as the stable one except that artifacts are published in private. So we need to promote git commits from a private repository to the public one then promote maven artifacts from a private maven repository to the public one.

The following sections assume you have prepared jenkinsci-cert/jenkins with security fixes and created a Maven staging repository as documented by the security team.

All steps need to be done twice: Once for weekly, once for LTS.

Preparation

1. Create a (origin) branch on jenkins-infra/release with a branch name corresponding to the specific release, e.g. security-2.287 or security-stable-2.303.2. Base them on the master (weekly) or stable-2.303 (LTS) branch, respectively.

2. In your fork, update the security environment file with the following (new) entries:

   1. RELEASE_GIT_BRANCH set to the jenkinsci-cert/jenkins release branch like security-stable-2.303

   2. MAVEN_REPOSITORY_NAME set to the maven repository name where we are going to publish staging maven artifacts, e.g. caravelli. This is also the source location used by the packaging job to build distribution packages.

   3. JENKINS_VERSION set to the final release version that will be packaged.

   4. RELEASELINE set to '-stable' for an LTS release, otherwise leave empty or undefined.

   5. Open a PR from your fork’s branch into the origin repository’s branch to allow review.

Staging (before release day)

To stage the Maven artifacts, trigger the generic Release job from the appropriate branch like security-stable-2.303.2.

To do that, follow these steps:

1. Force repository scan

2. Trigger the first build to have access to job parameter and immediately abort it

3. Trigger a job with the correct parameters

   1. RELEASE_PROFILE set to security

   2. RELEASE_GIT_BRANCH set to unused as we already define it in the release profile file, which overrides the job parameter

   3. MAVEN_REPOSITORY_NAME set to unused as we already define it in the release profile file, which overrides the job parameter

   4. VALIDATION_ENABLED set to true if the validation stage should run

Publishing (on release day)

1. To create and publish packages, trigger the generic Packaging job job from the appropriate branch like security-stable-2.303.2 with correct parameters

   1. RELEASE_PROFILE set to security

   2. RELEASE_GIT_BRANCH set to unused same reason as before

   3. MAVEN_REPOSITORY_NAME set to unused same reason as before

   4. MAVEN_REPOSITORY_PRODUCTION_NAME set to unused

   5. MAVEN_STAGING_REPOSITORY_PROMOTION_ENABLED set to false (manually done by publishing-tool in a parallel process)

   6. GIT_STAGING_REPOSITORY_PROMOTION_ENABLED set to false (manually merged by security team)

   7. VALIDATION_ENABLED set to true

Certificate

The Jenkins project uses a Digicert account provided by CDF to request code signing certificate. The release environment is designed to download a pkcs12 certificate from Azure key vault.

1. Request a code signing certificate from Digicert

2. Convert the code signing certificate from Digicert from p7b format to a pfx (with pkcs12) certificate which also includes the private key but not export password

3. Upload the pfx certificate to Azure Key Vault

4. Update the release environment credentials with appropriated password

Certificate Fields

   Country Name: US
   State: DE
   Organization: CDF Binary Project a Series of LF Projects, LLC
   Organization Unit: Jenkins Project
   Common Name: Jenkins

Generate a new code signing certificate private key and a certificate signing request:

openssl req -out jenkins-release.csr -new -newkey rsa:4096 -keyout jenkins-release.key

Show csr information

openssl req -text -noout -verify -in jenkins-release.csr

Show private key information

openssl rsa -in jenkins-release.key -check

Show certificate information

openssl x509 -in jenkins-release.crt -text -noout

Convert p7b to pkcs12

# Based from https://knowledge.digicert.com/solution/SO26449.html and https://github.com/jenkins-infra/release/blob/7a03f98eff839d4fed75ea96cf7bebbc963e3a91/README.adoc#certificate
# P7B to PFX: 1/2
openssl pkcs7 -print_certs -in digicert.p7b -out jenkins-release.crt
  ## Asks for the Export password, transmitted by Digicert from another channel
  ## Asks for the `jenkins-release.key` private key passphrase

# Check for the intermediate certificate attributes
openssl x509 -in jenkins-release.crt -text -noout

# P7B to PFX: 2/2
openssl pkcs12 -export -in jenkins-release.crt -inkey jenkins-release.key -out jenkins-release.pfx
  ## Asks for an Export password: do not set any (type enter only)
Enter Export Password: # Empty!!
Verifying - Enter Export Password: # Empty!!

Check PFX (pkcs12 format) attributes

openssl pkcs12 -info -in jenkins.pfx

Core Maintainers

More information about Jenkins Core maintainers and the different roles can be found in MAINTAINERS.

FAQ

The stage release failed and we already push commits during the maven release.

The problem here, is if we re-trigger the release job, we also update the release version again. So if the process fails on running mvn release:stage, then we can re-trigger it from inside the container as custom parameters are located in settings-release.xml.

kubectl get pods -n release # Looking for the correct jenkins agent name
kubectl exec -i -t -n release -c maven <pod_name alias jenkins agent> bash
cd /home/jenkins/agent/workspace/core_release_master/release
mvn -B -DstagingRepository=releases::https://repo.jenkins-ci.org/releases -s settings-release.xml --no-transfer-progress release:stage

Miscellaneous

• Jenkins docker images are describe from jenkinsci/docker repository and deployed to DockerHub as jenkins/jenkins

• pkg.jenkins.io, service to download Jenkins packages

• repo.jenkins-ci.org, Jenkins Maven repository

• mirrors.jenkins.io

• INFRA-910 - EPIC for the new Jenkins Core Release Environment

• docker-packaging, defines the docker image used to execute packaging scripts