Skip to content

Latest commit

 

History

History
292 lines (223 loc) · 14.1 KB

release-process.md

File metadata and controls

292 lines (223 loc) · 14.1 KB
Raw

Release Process

Create an Issue to Track the Release Process

To better coordinate the release process create an issue in the kserve/modelmesh-serving to keep track of all the steps that need to be done. Many of the checklists below are intended to be copied into the issue description so that the completed steps can be checked of while the release process is moving forward.

Also utilize the bi-weekly KServe community call to coordinate the release between KServe and ModelMesh or reach out to @dsun on the #kserve Kubeflow Slack channel.

Prepare the Release

Before starting the actual release process, make sure that the features and bug fixes that were designated to be part of the release are completed and fully tested.

Check the Security tab for each of the ModelMesh repositories and make sure all outstanding vulnerabilities were addressed:

Update the go dependency to github.com/kserve/kserve in go.mod and run go mod tidy. Since the KServe and ModelMesh releases are aligned, there should already be a v...-rc0 (or -rc1) of KServe before the ModelMesh release process is started. Use this opportunity to update other go dependencies as well. The dependencies should be updated on the main branch, before creating a new release branch.

Update the CRDs under config/crd/bases with the CRDs from the kserve/kserve repository (https://github.com/kserve/kserve/tree/master/config/crd) using their latest pre-release version:

  • config/crd/bases/serving.kserve.io_inferenceservices.yaml
  • config/crd/bases/serving.kserve.io_clusterservingruntimes.yaml
  • config/crd/bases/serving.kserve.io_servingruntimes.yaml

These changes should be committed to main, before creating a release branch.

Create Release Branches

A release branch should be substantially feature complete with respect to the intended release. Code that is committed to main may be merged or cherry-picked on to a release branch, but code that is directly committed to the release branch should be solely applicable to that release (and should not be committed back to main). In general, unless you're committing code that only applies to the release stream (for example, temporary hotfixes, backported security fixes, or image hashes), you should commit to main and then merge or cherry-pick to the release branch.

Create a release branch from main in the form of release-${MAJOR}.${MINOR}. Release branches serve several purposes:

  1. They allow a release wrangler or other developers to focus on a release without interrupting development on main,
  2. They allow developers to track the development of a release before a release candidate is declared,
  3. They simplify back porting critical bug fixes to a patch level release for a particular release stream (e.g., producing a v0.6.1 from release-0.6), when appropriate.

Create a (new) release-* branch in these 5 repositories:

Note: Technically, it is only required to create a release branch in the modelmesh-serving repository, where configuration files have to be modified with specific image tags corresponding to the new release being drafted. For the remaining repositories, a dedicated release branch can be created from the release tag at a later time, should the need arise to fix/patch a previous release.

Update Release Tags

It's generally a good idea to search the entire repo for strings of the old version number and replace them with the new, keeping in mind conflicts with other library version numbers. Most IDEs support searching an entire project using command-shift-f.

Some of the steps below need to be performed at least twice:

  • at least once for the release candidate(s) (v0.11.0-rc0, v0.11.0-rc1, ...) and
  • once more for the actual release (v0.11.0).

While creating a pre-release is not technically required, it is considered good practice. It allows other stakeholders to deploy and test the designated release, while keeping the door open to address any bugs before publishing a final release. This is especially useful when testing the new ModelMesh release in conjunction with KServe.

  1. Create new (pre-)release tags (v...-rc0) in these repositories:

    using the newly created release-* branches as target. This can be done by creating a draft release using the GitHub web interface and checking the "Set as a pre-release" option. The newly created tag should trigger the GitHub action to push the respective (pre-)release container images to DockerHub which are needed in the next step.

  2. Verify image tags were pushed to DockerHub:

  3. In this modelmesh-serving repository, on the release-* branch, update the container image tags to the corresponding release versions for:

    • kserve/modelmesh
    • kserve/modelmesh-controller
    • kserve/modelmesh-minio-examples
    • kserve/modelmesh-minio-dev-examples
    • kserve/modelmesh-runtime-adapter
    • kserve/rest-proxy

    The version tags should be updated in the following files:

    • .github/workflows/fvt-base.yml:
      • docker pull kserve/modelmesh:v...
      • docker pull kserve/modelmesh-minio-dev-examples:v...
      • docker pull kserve/modelmesh-minio-examples:v...
      • docker pull kserve/modelmesh-runtime-adapter:v...
      • docker pull kserve/rest-proxy:v...
    • config/default/config-defaults.yaml:
      • kserve/modelmesh
      • kserve/rest-proxy
      • kserve/modelmesh-runtime-adapter
    • config/dependencies/fvt.yaml:
      • image: kserve/modelmesh-minio-dev-examples:v...
      • image: kserve/modelmesh-minio-examples:v...
    • config/dependencies/quickstart.yaml:
      • image: kserve/modelmesh-minio-examples:v...
    • config/manager/kustomization.yaml: update the newTag to v...
    • docs/component-versions.md: update tags and repo links
      • ModelMesh Serving release
      • ModelMesh
      • ModelMesh Runtime Adapter
      • REST Proxy
    • docs/install/install-script.md: update the RELEASE variable in the Installation section to the new release-* branch name and remove the note pointing to the (old) release-* branch
    • docs/quickstart.md: update the RELEASE variable in the "Clone the ModelMesh repository" section to the new release-* branch and remove the note of caution in the introduction above
    • scripts/setup_user_namespaces.sh: change the modelmesh_release version

    You can copy the checklist above into the PR description in the next step.

  4. Submit your PR to the release-* branch that was created earlier and wait for it to merge.

  5. Update the following files in the main branch, replacing all occurrences pointing to the old release-* branch or the previous release version tag (e.g. v0.11.0) with the new release branch name or new version tags. Submit them in a PR to main, and wait for that PR to be merged:

    • docs/install/install-script.md
    • docs/component-versions.md
    • docs/quickstart.md
    • scripts/setup_user_namespaces.sh

Generate Release Artifacts and Publish the Release

  1. Generate the release manifests on the release-* branch:

    kustomize build config/runtimes --load-restrictor LoadRestrictionsNone > modelmesh-runtimes.yaml
cp config/dependencies/quickstart.yaml modelmesh-quickstart-dependencies.yaml

    If you see Error: unknown flag: --load-restrictor upgrade your kustomize version to 4.x.

  2. Generate config archive on the release-* branch. The scriptlet below automatically determines the release version and chooses the version of the tar command for either Linux or macOS. Verify the correct release VERSION was found.

    VERSION=$( grep -o -E "newTag: .*$" config/manager/kustomization.yaml | sed 's/newTag: //' )
TAR_FILE="config-${VERSION}.tar.gz"

echo "Release: ${VERSION}"

if $(tar --version | grep -q 'bsd'); then
  tar -zcvf ${TAR_FILE} -s /config/config-${VERSION}/ config/;
else
  tar -zcvf ${TAR_FILE} config/ --transform s/config/config-${VERSION}/;
fi

  3. Create a new tag on the release-* branch and push it to GitHub using the commands below, or, create a new tag in the next step using the GitHub UI. The new kserve/modelmesh-controller image will be published via GitHub Actions.

    git tag $VERSION
git push upstream $VERSION

echo https://github.com/kserve/modelmesh-serving/releases/new?tag=${VERSION}

  4. Create the new release in the GitHub UI from the release-* branch (or from the tag created in the previous step). Enter the release tag value (e.g. v0.11.0) in the "Release title" field and upload the generated installation manifests ("Release assets") in the "Attach binaries ..." section. Click the "Generate release notes" button which will generate the release description.

    Note, if you generated a pre-release (e.g. v0.11.0-rc0) then copy the release notes from that and remove them from the pre-release description and revise accordingly.

    https://github.com/kserve/modelmesh-serving/releases/new

  5. Compare the release and release artifacts to those of previous releases to make sure nothing was missed.

  6. Once the release as been published (a new tag has been pushed), verify the check results by clicking on the check mark (✓) next to the latest commit on the release-* branch.

  7. Verify that the newly released version of the modelmesh-controller was pushed to DockerHub.

Update the KServe Helm Charts

Fork and clone the kserve/kserve repository and update all references to the old ModelMesh versions. At the time of the v0.10.0 release the following files needed to be updated.

  • charts/kserve-resources/values.yaml
  • hack/install_kserve_mm.sh

Furthermore, the helm charts under charts/kserve-resources/templates which are used to install ModelMesh as part of KServe need to be updated with the changes in the respective manifests from the kserve/modelmesh-serving repository found in the config folder.

For reference, for the v0.10.0 release the following charts in the kserve repo had to be updated:

  • charts/kserve-resources/templates/clusterrole.yaml
  • charts/kserve-resources/templates/clusterservingruntimes.yaml
  • charts/kserve-resources/templates/configmap.yaml
  • charts/kserve-resources/templates/deployment.yaml

For the v0.9.0 release the following charts had to be updated:

  • charts/kserve/crds/serving.kserve.io_predictor.yaml
  • charts/kserve/templates/clusterrole.yaml
  • charts/kserve/templates/configmap.yaml
  • charts/kserve/templates/deployment.yaml
  • charts/kserve/templates/networkpolicy.yaml
  • charts/kserve/templates/rolebinding.yaml
  • charts/kserve/templates/servingruntimes.yaml

Update the KServe Website

In the kserve/website repository, update all reference to the previous ModelMesh release. As of v0.10.0, docs/admin/modelmesh.md was the only Markdown file to be updated.

Release Blog

Work with Dan Sun on a joint release blog.

For reference, here are a few examples of previous release blogs featuring ModelMesh:

And the corresponding PRs to illustrate the process and the participants: