Add nightly release pipeline 🌙 #1274
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,226 @@ | ||
| apiVersion: tekton.dev/v1alpha1 | ||
|
There was a problem hiding this comment. There are a few differences between publish and publish-nightly, but not so many, so it would be nice eventually to have them converge to a single Task executed with different inputs. One difference that might be difficult to overcome is the version of the ko-ci image, which is |
||
| kind: Task | ||
| metadata: | ||
| name: publish-tekton-pipelines-nightly | ||
| spec: | ||
| inputs: | ||
| resources: | ||
| - name: source | ||
| type: git | ||
| targetPath: go/src/github.com/tektoncd/pipeline | ||
| - name: bucket | ||
| type: storage | ||
| params: | ||
| # TODO(triggers#87) This Task has no verisonTag parameter (like publish-tekton-pipelines) because Prow does not allow the value to be provided dynamically | ||
| - name: imageRegistry | ||
| description: TODO(#569) This is a hack to make it easy for folks to switch the registry being used by the many many image outputs | ||
| - name: pathToProject | ||
| description: The path to the folder in the go/src dir that contains the project, which is used by `ko` to name the resulting images | ||
| outputs: | ||
| resources: | ||
| - name: bucket | ||
| type: storage | ||
| - name: builtBaseImage | ||
| type: image | ||
| - name: builtEntrypointImage | ||
| type: image | ||
| - name: builtKubeconfigWriterImage | ||
| type: image | ||
| - name: builtCredsInitImage | ||
| type: image | ||
| - name: builtGitInitImage | ||
| type: image | ||
| - name: builtNopImage | ||
| type: image | ||
| - name: builtBashImage | ||
| type: image | ||
| - name: builtGsutilImage | ||
| type: image | ||
| - name: builtControllerImage | ||
| type: image | ||
| - name: builtWebhookImage | ||
| type: image | ||
| - name: builtDigestExporterImage | ||
| type: image | ||
| - name: builtPullRequestInitImage | ||
| type: image | ||
| - name: builtGcsFetcherImage | ||
| type: image | ||
| steps: | ||
|
|
||
| - name: build-push-base-images | ||
| image: gcr.io/kaniko-project/executor:v0.9.0 | ||
| command: | ||
| - /kaniko/executor | ||
| args: | ||
| - --dockerfile=/workspace/go/src/github.com/tektoncd/pipeline/images/Dockerfile | ||
| - --destination=${inputs.params.imageRegistry}/${inputs.params.pathToProject}/${outputs.resources.builtBaseImage.url} | ||
| - --context=/workspace/go/src/github.com/tektoncd/pipeline | ||
|
|
||
| volumeMounts: | ||
| - name: gcp-secret | ||
| mountPath: /secret | ||
| env: | ||
| - name: GOOGLE_APPLICATION_CREDENTIALS | ||
| value: /secret/release.json | ||
|
|
||
| - name: create-ko-yaml | ||
| image: busybox | ||
| command: | ||
| - /bin/sh | ||
| args: | ||
| - -ce | ||
| - | | ||
| set -e | ||
| set -x | ||
|
|
||
| cat <<EOF > /workspace/go/src/github.com/tektoncd/pipeline/.ko.yaml | ||
| # By default ko will build images on top of distroless | ||
| baseImageOverrides: | ||
| # Use the images we just built as base images | ||
| $(inputs.params.pathToProject)/$(outputs.resources.builtCredsInitImage.url): ${inputs.params.imageRegistry}/${inputs.params.pathToProject}/build-base:latest | ||
| $(inputs.params.pathToProject)/$(outputs.resources.builtGitInitImage.url): ${inputs.params.imageRegistry}/${inputs.params.pathToProject}/build-base:latest | ||
|
|
||
| $(inputs.params.pathToProject)/$(outputs.resources.builtBashImage.url): busybox # image should have shell in $PATH | ||
| $(inputs.params.pathToProject)/$(outputs.resources.builtEntrypointImage.url): busybox # image should have shell in $PATH | ||
| $(inputs.params.pathToProject)/$(outputs.resources.builtGsutilImage.url): google/cloud-sdk:alpine # image should have gsutil in $PATH | ||
| EOF | ||
|
|
||
| cat /workspace/go/src/github.com/tektoncd/pipeline/.ko.yaml | ||
|
|
||
| - name: ensure-release-dirs-exist | ||
| image: busybox | ||
| command: ["mkdir"] | ||
| args: | ||
| - "-p" | ||
| - "/workspace/bucket/latest/" | ||
| - "/workspace/bucket/previous/" | ||
|
|
||
| - name: run-ko | ||
| # TODO(#639) we should be able to use the image built by an upstream Task here instead of hardcoding | ||
| image: gcr.io/tekton-nightly/ko-ci | ||
| env: | ||
| - name: KO_DOCKER_REPO | ||
| value: ${inputs.params.imageRegistry} | ||
| - name: GOPATH | ||
| value: /workspace/go | ||
| - name: CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE | ||
| value: /secret/release.json | ||
| command: | ||
| - /bin/sh | ||
| args: | ||
| - -ce | ||
| - | | ||
| set -e | ||
| set -x | ||
|
|
||
| # Auth with CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE | ||
| gcloud auth configure-docker | ||
|
|
||
| # ko requires this variable to be set in order to set image creation timestamps correctly https://github.com/google/go-containerregistry/pull/146 | ||
| export SOURCE_DATE_EPOCH=`date +%s` | ||
|
|
||
| # Change to directory with our .ko.yaml | ||
| cd /workspace/go/src/github.com/tektoncd/pipeline | ||
|
|
||
| # Publish images and create release.yaml | ||
| ko resolve --preserve-import-paths -f /workspace/go/src/github.com/tektoncd/pipeline/config/ > /workspace/bucket/latest/release.yaml | ||
| volumeMounts: | ||
| - name: gcp-secret | ||
| mountPath: /secret | ||
|
|
||
| - name: generate-release-version | ||
| image: alpine/git | ||
| command: | ||
| - /bin/sh | ||
| args: | ||
| - -ce | ||
| - | | ||
| set -e | ||
| set -x | ||
|
|
||
| # Can't use workingDir due to #1267 | ||
| cd /workspace/go/src/github.com/tektoncd/pipeline | ||
|
|
||
| # Nightly releases use vYYYYMMDD-commit | ||
| COMMIT=$(git rev-parse HEAD | cut -c 1-10) | ||
| DATE=$(date +"%Y%m%d") | ||
| VERSION_TAG="$DATE-$COMMIT" | ||
|
|
||
| echo $VERSION_TAG > "/builder/home/version" | ||
|
|
||
| - name: copy-to-tagged-bucket | ||
| image: busybox | ||
| command: | ||
| - /bin/sh | ||
| args: | ||
| - -ce | ||
| - | | ||
| set -e | ||
| set -x | ||
|
|
||
| # Can't use workingDir due to #1267 | ||
| cd /workspace/bucket | ||
|
|
||
| # TODO(https://github.com/tektoncd/triggers/issues/87) if the versionTag could be generated dynamically, we could use the same Task for nightly + official releases | ||
| export VERSION_TAG="$(cat /builder/home/version)" | ||
|
|
||
| mkdir -p /workspace/bucket/previous/$VERSION_TAG/ | ||
| cp /workspace/bucket/latest/release.yaml /workspace/bucket/previous/$VERSION_TAG/release.yaml | ||
|
|
||
| - name: tag-images | ||
|
There was a problem hiding this comment. Kaniko (the latest version at least) supports specifying multiple destinations for a single build command, meaning that we could build and tag the images in the same step. It might be that the code would be less readable though. |
||
| image: google/cloud-sdk | ||
| command: | ||
| - /bin/bash | ||
| args: | ||
| - -ce | ||
| - | | ||
| set -e | ||
| set -x | ||
|
|
||
| # TODO(https://github.com/tektoncd/triggers/issues/87) if the versionTag could be generated dynamically, we could use the same Task for nightly + official releases | ||
| export VERSION_TAG="$(cat /builder/home/version)" | ||
|
|
||
| REGIONS=(us eu asia) | ||
| IMAGES=( | ||
| ${inputs.params.imageRegistry}/${inputs.params.pathToProject}/${outputs.resources.builtEntrypointImage.url} | ||
| ${inputs.params.imageRegistry}/${inputs.params.pathToProject}/${outputs.resources.builtKubeconfigWriterImage.url} | ||
| ${inputs.params.imageRegistry}/${inputs.params.pathToProject}/${outputs.resources.builtCredsInitImage.url} | ||
| ${inputs.params.imageRegistry}/${inputs.params.pathToProject}/${outputs.resources.builtGitInitImage.url} | ||
| ${inputs.params.imageRegistry}/${inputs.params.pathToProject}/${outputs.resources.builtNopImage.url} | ||
| ${inputs.params.imageRegistry}/${inputs.params.pathToProject}/${outputs.resources.builtBashImage.url} | ||
| ${inputs.params.imageRegistry}/${inputs.params.pathToProject}/${outputs.resources.builtGsutilImage.url} | ||
| ${inputs.params.imageRegistry}/${inputs.params.pathToProject}/${outputs.resources.builtControllerImage.url} | ||
| ${inputs.params.imageRegistry}/${inputs.params.pathToProject}/${outputs.resources.builtWebhookImage.url} | ||
| ${inputs.params.imageRegistry}/${inputs.params.pathToProject}/${outputs.resources.builtDigestExporterImage.url} | ||
| ${inputs.params.imageRegistry}/${inputs.params.pathToProject}/${outputs.resources.builtPullRequestInitImage.url} | ||
| ${inputs.params.imageRegistry}/${inputs.params.pathToProject}/${outputs.resources.builtGcsFetcherImage.url} | ||
| ) | ||
| # Parse the built images from the release.yaml generated by ko | ||
| BUILT_IMAGES=( $(/workspace/go/src/github.com/tektoncd/pipeline/tekton/koparse/koparse.py --path /workspace/bucket/latest/release.yaml --base ${inputs.params.imageRegistry}/${inputs.params.pathToProject} --images ${IMAGES[@]}) ) | ||
|
|
||
| # Auth with account credentials | ||
| gcloud auth activate-service-account --key-file=/secret/release.json | ||
|
|
||
| # Tag the images and put them in all the regions | ||
| for IMAGE in "${BUILT_IMAGES[@]}" | ||
| do | ||
| IMAGE_WITHOUT_SHA=${IMAGE%%@*} | ||
| gcloud -q container images add-tag ${IMAGE} ${IMAGE_WITHOUT_SHA}:latest | ||
| gcloud -q container images add-tag ${IMAGE} ${IMAGE_WITHOUT_SHA}:$VERSION_TAG | ||
| for REGION in "${REGIONS[@]}" | ||
| do | ||
| for TAG in "latest" $VERSION_TAG | ||
| do | ||
| gcloud -q container images add-tag ${IMAGE} ${REGION}.${IMAGE_WITHOUT_SHA}:$TAG | ||
| done | ||
| done | ||
| done | ||
| volumeMounts: | ||
| - name: gcp-secret | ||
| mountPath: /secret | ||
|
|
||
| volumes: | ||
| - name: gcp-secret | ||
| secret: | ||
| secretName: release-secret | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This could be specified in the
PipelineRunwithPodTemplate(that said it's not supported yet bytkn…)