To better coordinate the release process create an issue in the kserve/modelmesh-serving to keep track of all the steps that need to be done. Many of the checklists below are intended to be copied into the issue description so that the completed steps can be checked of while the release process is moving forward.
Also utilize the bi-weekly KServe community call
to coordinate the release between KServe and ModelMesh or reach out to @dsun
on the #kserve
Kubeflow Slack channel.
Before starting the actual release process, make sure that the features and bug fixes that were designated to be part of the release are completed and fully tested.
Check the Security tab for each of the ModelMesh repositories and make sure all outstanding vulnerabilities were addressed:
Update the go
dependency to github.com/kserve/kserve
in go.mod
and run
go mod tidy
. Since the KServe and ModelMesh releases are aligned, there should
already be a v...-rc0
(or -rc1
) of KServe before the ModelMesh release process
is started. Use this opportunity to update other go
dependencies as well.
The dependencies should be updated on the main
branch, before creating a new
release branch.
Update the CRDs under config/crd/bases
with the CRDs from the kserve/kserve
repository (https://github.com/kserve/kserve/tree/master/config/crd
) using their
latest pre-release version:
-
config/crd/bases/serving.kserve.io_inferenceservices.yaml
-
config/crd/bases/serving.kserve.io_clusterservingruntimes.yaml
-
config/crd/bases/serving.kserve.io_servingruntimes.yaml
These changes should be committed to main
, before creating a release branch.
A release branch should be substantially feature complete with respect to the
intended release. Code that is committed to main
may be merged or cherry-picked
on to a release branch, but code that is directly committed to the release branch
should be solely applicable to that release (and should not be committed back to
main
). In general, unless you're committing code that only applies to the release
stream (for example, temporary hotfixes, backported security fixes, or image hashes),
you should commit to main
and then merge or cherry-pick to the release branch.
Create a release branch from main
in the form of release-${MAJOR}.${MINOR}
.
Release branches serve several purposes:
- They allow a release wrangler or other developers to focus on a release without
interrupting development on
main
, - They allow developers to track the development of a release before a release candidate is declared,
- They simplify back porting critical bug fixes to a patch level release for a
particular release stream (e.g., producing a
v0.6.1
fromrelease-0.6
), when appropriate.
Create a (new) release-*
branch in these 5 repositories:
Note: Technically, it is only required to create a release branch in the
modelmesh-serving
repository,
where configuration files have to be modified with specific image tags corresponding
to the new release being drafted. For the remaining repositories, a dedicated release
branch can be created from the release tag at a later time, should the need arise to
fix/patch a previous release.
It's generally a good idea to search the entire repo for strings of the old version
number and replace them with the new, keeping in mind conflicts with other library
version numbers. Most IDEs support searching an entire project using command
-shift
-f
.
Some of the steps below need to be performed at least twice:
- at least once for the release candidate(s) (
v0.11.0-rc0
,v0.11.0-rc1
, ...) and - once more for the actual release (
v0.11.0
).
While creating a pre-release is not technically required, it is considered good practice. It allows other stakeholders to deploy and test the designated release, while keeping the door open to address any bugs before publishing a final release. This is especially useful when testing the new ModelMesh release in conjunction with KServe.
-
Create new (pre-)release tags (
v...-rc0
) in these repositories:using the newly created
release-*
branches as target. This can be done by creating a draft release using the GitHub web interface and checking the "Set as a pre-release" option. The newly created tag should trigger the GitHub action to push the respective (pre-)release container images to DockerHub which are needed in the next step. -
Verify image tags were pushed to DockerHub:
-
In this
modelmesh-serving
repository, on therelease-*
branch, update the container image tags to the corresponding release versions for:kserve/modelmesh
kserve/modelmesh-controller
kserve/modelmesh-minio-examples
kserve/modelmesh-minio-dev-examples
kserve/modelmesh-runtime-adapter
kserve/rest-proxy
The version tags should be updated in the following files:
-
.github/workflows/fvt-base.yml
:-
docker pull kserve/modelmesh:v...
-
docker pull kserve/modelmesh-minio-dev-examples:v...
-
docker pull kserve/modelmesh-minio-examples:v...
-
docker pull kserve/modelmesh-runtime-adapter:v...
-
docker pull kserve/rest-proxy:v...
-
-
config/default/config-defaults.yaml
:-
kserve/modelmesh
-
kserve/rest-proxy
-
kserve/modelmesh-runtime-adapter
-
-
config/dependencies/fvt.yaml
:-
image: kserve/modelmesh-minio-dev-examples:v...
-
image: kserve/modelmesh-minio-examples:v...
-
-
config/dependencies/quickstart.yaml
:-
image: kserve/modelmesh-minio-examples:v...
-
-
config/manager/kustomization.yaml
: update thenewTag
tov...
-
docs/component-versions.md
: update tags and repo links- ModelMesh Serving release
- ModelMesh
- ModelMesh Runtime Adapter
- REST Proxy
-
docs/install/install-script.md
: update theRELEASE
variable in theInstallation
section to the newrelease-*
branch name and remove the note pointing to the (old)release-*
branch -
docs/quickstart.md
: update theRELEASE
variable in the "Clone the ModelMesh repository" section to the newrelease-*
branch and remove the note of caution in the introduction above -
scripts/setup_user_namespaces.sh
: change themodelmesh_release
version
You can copy the checklist above into the PR description in the next step.
-
Submit your PR to the
release-*
branch that was created earlier and wait for it to merge. -
Update the following files in the
main
branch, replacing all occurrences pointing to the oldrelease-*
branch or the previous release version tag (e.g.v0.11.0
) with the new release branch name or new version tags. Submit them in a PR tomain
, and wait for that PR to be merged:-
docs/install/install-script.md
-
docs/component-versions.md
-
docs/quickstart.md
-
scripts/setup_user_namespaces.sh
-
-
Generate the release manifests on the
release-*
branch:kustomize build config/runtimes --load-restrictor LoadRestrictionsNone > modelmesh-runtimes.yaml cp config/dependencies/quickstart.yaml modelmesh-quickstart-dependencies.yaml
If you see
Error: unknown flag: --load-restrictor
upgrade yourkustomize
version to 4.x. -
Generate config archive on the
release-*
branch. The scriptlet below automatically determines the release version and chooses the version of thetar
command for either Linux or macOS. Verify the correct releaseVERSION
was found.VERSION=$( grep -o -E "newTag: .*$" config/manager/kustomization.yaml | sed 's/newTag: //' ) TAR_FILE="config-${VERSION}.tar.gz" echo "Release: ${VERSION}" if $(tar --version | grep -q 'bsd'); then tar -zcvf ${TAR_FILE} -s /config/config-${VERSION}/ config/; else tar -zcvf ${TAR_FILE} config/ --transform s/config/config-${VERSION}/; fi
-
Create a new tag on the
release-*
branch and push it to GitHub using the commands below, or, create a new tag in the next step using the GitHub UI. The newkserve/modelmesh-controller
image will be published via GitHub Actions.git tag $VERSION git push upstream $VERSION echo https://github.com/kserve/modelmesh-serving/releases/new?tag=${VERSION}
-
Create the new release in the GitHub UI from the
release-*
branch (or from the tag created in the previous step). Enter the release tag value (e.g.v0.11.0
) in the "Release title" field and upload the generated installation manifests ("Release assets") in the "Attach binaries ..." section. Click the "Generate release notes" button which will generate the release description.Note, if you generated a pre-release (e.g.
v0.11.0-rc0
) then copy the release notes from that and remove them from the pre-release description and revise accordingly. -
Compare the release and release artifacts to those of previous releases to make sure nothing was missed.
-
Once the release as been published (a new tag has been pushed), verify the check results by clicking on the check mark (✓) next to the latest commit on the
release-*
branch. -
Verify that the newly released version of the
modelmesh-controller
was pushed to DockerHub.
Fork and clone the kserve/kserve
repository
and update all references to the old ModelMesh versions. At the time of the v0.10.0
release the following files needed to be updated.
charts/kserve-resources/values.yaml
hack/install_kserve_mm.sh
Furthermore, the helm
charts under charts/kserve-resources/templates
which are
used to install ModelMesh as part of KServe need to be updated with the changes
in the respective manifests from the kserve/modelmesh-serving
repository found
in the config
folder.
For reference, for the v0.10.0
release the following charts in the kserve
repo
had to be updated:
charts/kserve-resources/templates/clusterrole.yaml
charts/kserve-resources/templates/clusterservingruntimes.yaml
charts/kserve-resources/templates/configmap.yaml
charts/kserve-resources/templates/deployment.yaml
For the v0.9.0
release the following charts had to be updated:
charts/kserve/crds/serving.kserve.io_predictor.yaml
charts/kserve/templates/clusterrole.yaml
charts/kserve/templates/configmap.yaml
charts/kserve/templates/deployment.yaml
charts/kserve/templates/networkpolicy.yaml
charts/kserve/templates/rolebinding.yaml
charts/kserve/templates/servingruntimes.yaml
In the kserve/website
repository, update all reference to the previous ModelMesh
release. As of v0.10.0
, docs/admin/modelmesh.md
was the only Markdown file to
be updated.
Work with Dan Sun on a joint release blog.
For reference, here are a few examples of previous release blogs featuring ModelMesh:
And the corresponding PRs to illustrate the process and the participants: