Skip to content

Commit

Permalink
pipeline level finally - implementation
Browse files Browse the repository at this point in the history
We can now specify a list of tasks needs to be executed just before
pipeline exits (either after finishing all non-final tasks successfully or after
a single failure)

Most useful for tasks such as report test results, cleanup cluster resources, etc

```
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: pipeline-with-final-tasks
spec:
  tasks:
    - name: pre-work
      taskRef:
        Name: some-pre-work
    - name: unit-test
      taskRef:
        Name: run-unit-test
      runAfter:
        - pre-work
    - name: integration-test
      taskRef:
        Name: run-integration-test
      runAfter:
        - unit-test
  finally:
    - name: cleanup-test
      taskRef:
        Name: cleanup-cluster
    - name: report-results
      taskRef:
        Name: report-test-results
```
  • Loading branch information
pritidesai committed Jun 25, 2020
1 parent 44f22a0 commit 3e22d0e
Show file tree
Hide file tree
Showing 9 changed files with 1,625 additions and 78 deletions.
191 changes: 182 additions & 9 deletions docs/pipelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ weight: 3
- [Configuring execution results at the `Pipeline` level](#configuring-execution-results-at-the-pipeline-level)
- [Configuring the `Task` execution order](#configuring-the-task-execution-order)
- [Adding a description](#adding-a-description)
- [Adding `Finally` to the `Pipeline` (Preview)](#adding-finally-to-the-pipeline-preview)
- [Adding `Finally` to the `Pipeline`](#adding-finally-to-the-pipeline)
- [Code examples](#code-examples)

## Overview
Expand Down Expand Up @@ -530,10 +530,7 @@ In particular:

The `description` field is an optional field and can be used to provide description of the `Pipeline`.

## Adding `Finally` to the `Pipeline` (Preview)

_Finally type is available in the `Pipeline` but functionality is in progress. Final tasks are can be specified and
are validated but not executed yet._
## Adding `Finally` to the `Pipeline`

You can specify a list of one or more final tasks under `finally` section. Final tasks are guaranteed to be executed
in parallel after all `PipelineTasks` under `tasks` have completed regardless of success or error. Final tasks are very
Expand All @@ -553,10 +550,186 @@ spec:
Name: cleanup
```

_[PR #2661](https://github.com/tektoncd/pipeline/pull/2661) is implementing this new functionality by adding support to enable
final tasks along with workspaces and parameters. `PipelineRun` status is being updated to include execution status of
final tasks i.e. `PipelineRun` status is set to success or failure depending on execution of `PipelineTasks`, this status
remains same when all final tasks finishes successfully but is set to failure if any of the final tasks fail._
### Specifying `Workspaces` in Final Tasks

Finally tasks can specify [workspaces](workspaces.md) which `PipelineTasks` might have utilized
e.g. a mount point for credentials held in Secrets. To support that requirement, you can specify one or more
`Workspaces` in the `workspaces` field for the final tasks similar to `tasks`.

```yaml
spec:
resources:
- name: app-git
type: git
workspaces:
- name: shared-workspace
tasks:
- name: clone-app-source
taskRef:
name: clone-app-repo-to-workspace
workspaces:
- name: shared-workspace
workspace: shared-workspace
resources:
inputs:
- name: app-git
resource: app-git
finally:
- name: cleanup-workspace
taskRef:
name: cleanup-workspace
workspaces:
- name: shared-workspace
workspace: shared-workspace
```

### Specifying `Parameters` in Final Tasks

Similar to `tasks`, you can specify [`Parameters`](tasks.md#specifying-parameters) in final tasks:

```yaml
spec:
tasks:
- name: tests
taskRef:
Name: integration-test
finally:
- name: report-results
taskRef:
Name: report-results
params:
- name: url
value: "someURL"
```

### `PipelineRun` Status with `finally`

With `finally`, `PipelineRun` status is calculated based on `PipelineTasks` under `tasks` section and final tasks.

Without `finally`:

| `PipelineTasks` under `tasks` | `PipelineRun` status | Reason |
| ----------------------------- | -------------------- | ------ |
| all `PipelineTasks` successful | `true` | `Succeeded` |
| one or more `PipelineTasks` [skipped](conditions.md) and rest successful | `true` | `Completed` |
| single failure of `PipelineTask` | `false` | `failed` |

With `finally`:

| `PipelineTasks` under `tasks` | Final Tasks | `PipelineRun` status | Reason |
| ----------------------------- | ----------- | -------------------- | ------ |
| all `PipelineTask` successful | all final tasks successful | `true` | `Succeeded` |
| all `PipelineTask` successful | one or more failure of final tasks | `false` | `Failed` |
| one or more `PipelineTask` [skipped](conditions.md) and rest successful | all final tasks successful | `true` | `Completed` |
| one or more `PipelineTask` [skipped](conditions.md) and rest successful | one or more failure of final tasks | `false` | `Failed` |
| single failure of `PipelineTask` | all final tasks successful | `false` | `failed` |
| single failure of `PipelineTask` | one or more failure of final tasks | `false` | `failed` |

Overall, `PipelineRun` state transitioning is explained below for respective scenarios:

* All `PipelineTask` and final tasks are successful: `Started` -> `Running` -> `Succeeded`
* At least one `PipelineTask` skipped and rest successful: `Started` -> `Running` -> `Completed`
* One `PipelineTask` failed / one or more final tasks failed: `Started` -> `Running` -> `Failed`

Please refer to the [table](pipelineruns.md#monitoring-execution-status) under Monitoring Execution Status to learn about
what kind of events are triggered based on the `Pipelinerun` status.

### Known Limitations

### Specifying `Resources` in Final Tasks

Similar to `tasks`, you can use [PipelineResources](#specifying-resources) as inputs and outputs for
final tasks in the Pipeline. The only difference here is, final tasks with an input resource can not have a `from` clause
like a `PipelineTask` from `tasks` section. For example:

```yaml
spec:
tasks:
- name: tests
taskRef:
Name: integration-test
resources:
inputs:
- name: source
resource: tektoncd-pipeline-repo
outputs:
- name: workspace
resource: my-repo
finally:
- name: clear-workspace
taskRef:
Name: clear-workspace
resources:
inputs:
- name: workspace
resource: my-repo
from: #invalid
- tests
```

### Cannot configure the Final Task execution order

It's not possible to configure or modify the execution order of the final tasks. Unlike `Tasks` in a `Pipeline`,
all final tasks run simultaneously and start executing once all `PipelineTasks` under `tasks` have settled which means
no `runAfter` can be specified in final tasks.

### Cannot specify execution `Conditions` in Final Tasks

`Tasks` in a `Pipeline` can be configured to run only if some conditions are satisfied using `conditions`. But the
final tasks are guaranteed to be executed after all `PipelineTasks` therefore no `conditions` can be specified in
final tasks.

#### Cannot configure `Task` execution results with `finally`

Final tasks can not be configured to consume `Results` of `PipelineTask` from `tasks` section i.e. the following
example is not supported right now but we are working on adding support for the same (tracked in issue
[#2557](https://github.com/tektoncd/pipeline/issues/2557)).

```yaml
spec:
tasks:
- name: count-comments-before
taskRef:
Name: count-comments
- name: add-comment
taskRef:
Name: add-comment
- name: count-comments-after
taskRef:
Name: count-comments
finally:
- name: check-count
taskRef:
Name: check-count
params:
- name: before-count
value: $(tasks.count-comments-before.results.count) #invalid
- name: after-count
value: $(tasks.count-comments-after.results.count) #invalid
```

#### Cannot configure `Pipeline` result with `finally`

Final tasks can emit `Results` but results emitted from the final tasks can not be configured in the
[Pipeline Results](#configuring-execution-results-at-the-pipeline-level). We are working on adding support for this
(tracked in issue [#2710](https://github.com/tektoncd/pipeline/issues/2710)).

```yaml
results:
- name: comment-count-validate
value: $(finally.check-count.results.comment-count-validate)
```

In this example, `PipelineResults` is set to:

```
"pipelineResults": [
{
"name": "comment-count-validate",
"value": "$(finally.check-count.results.comment-count-validate)"
}
],
```

## Code examples

Expand Down
187 changes: 187 additions & 0 deletions examples/v1beta1/pipelineruns/pipelinerun-with-final-tasks.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,187 @@
# Copied from https://github.com/tektoncd/catalog/blob/v1beta1/git/git-clone.yaml :(
# This can be deleted after we add support to refer to the remote Task in a registry (Issue #1839) or
# add support for referencing task in git directly (issue #2298)
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: git-clone-from-catalog
spec:
workspaces:
- name: output
description: The git repo will be cloned onto the volume backing this workspace
params:
- name: url
description: git url to clone
type: string
- name: revision
description: git revision to checkout (branch, tag, sha, ref…)
type: string
default: master
- name: refspec
description: (optional) git refspec to fetch before checking out revision
default: ""
- name: submodules
description: defines if the resource should initialize and fetch the submodules
type: string
default: "true"
- name: depth
description: performs a shallow clone where only the most recent commit(s) will be fetched
type: string
default: "1"
- name: sslVerify
description: defines if http.sslVerify should be set to true or false in the global git config
type: string
default: "true"
- name: subdirectory
description: subdirectory inside the "output" workspace to clone the git repo into
type: string
default: ""
- name: deleteExisting
description: clean out the contents of the repo's destination directory (if it already exists) before trying to clone the repo there
type: string
default: "false"
- name: httpProxy
description: git HTTP proxy server for non-SSL requests
type: string
default: ""
- name: httpsProxy
description: git HTTPS proxy server for SSL requests
type: string
default: ""
- name: noProxy
description: git no proxy - opt out of proxying HTTP/HTTPS requests
type: string
default: ""
results:
- name: commit
description: The precise commit SHA that was fetched by this Task
steps:
- name: clone
image: gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init:v0.12.1
script: |
CHECKOUT_DIR="$(workspaces.output.path)/$(params.subdirectory)"
cleandir() {
# Delete any existing contents of the repo directory if it exists.
#
# We don't just "rm -rf $CHECKOUT_DIR" because $CHECKOUT_DIR might be "/"
# or the root of a mounted volume.
if [[ -d "$CHECKOUT_DIR" ]] ; then
# Delete non-hidden files and directories
rm -rf "$CHECKOUT_DIR"/*
# Delete files and directories starting with . but excluding ..
rm -rf "$CHECKOUT_DIR"/.[!.]*
# Delete files and directories starting with .. plus any other character
rm -rf "$CHECKOUT_DIR"/..?*
fi
}
if [[ "$(params.deleteExisting)" == "true" ]] ; then
cleandir
fi
test -z "$(params.httpProxy)" || export HTTP_PROXY=$(params.httpProxy)
test -z "$(params.httpsProxy)" || export HTTPS_PROXY=$(params.httpsProxy)
test -z "$(params.noProxy)" || export NO_PROXY=$(params.noProxy)
/ko-app/git-init \
-url "$(params.url)" \
-revision "$(params.revision)" \
-refspec "$(params.refspec)" \
-path "$CHECKOUT_DIR" \
-sslVerify="$(params.sslVerify)" \
-submodules="$(params.submodules)" \
-depth "$(params.depth)"
cd "$CHECKOUT_DIR"
RESULT_SHA="$(git rev-parse HEAD | tr -d '\n')"
EXIT_CODE="$?"
if [ "$EXIT_CODE" != 0 ]
then
exit $EXIT_CODE
fi
# Make sure we don't add a trailing newline to the result!
echo -n "$RESULT_SHA" > $(results.commit.path)
---

# Task to cleanup shared workspace
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: cleanup-workspace
spec:
workspaces:
# Shared workspace where git repo is cloned
- name: source
steps:
- name: check-application-dir-has-source
image: ubuntu
script: |
if [ ! -d "$(workspaces.source.path)/application/" ]; then
echo "Something went wrong and could not find application source under $(workspaces.source.path)/application/"
exit 1
fi
- name: cleanup-workspace
image: ubuntu
script: |
rm -rf $(workspaces.source.path)/application/
- name: verify-application-dir-has-gone
image: ubuntu
script: |
if [ -d "$(workspaces.source.path)/application/" ]; then
echo "Something went wrong cleaning up and the application source still exists under $(workspaces.source.path)/application/"
exit 1
fi
---

# Pipeline to clone repo into shared workspace and cleanup the workspace after done
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
name: clone-cleanup-workspace
spec:
workspaces:
# common workspace where git repo is cloned and needs to be cleanup after done
- name: git-source
tasks:
# Clone app repo to workspace
- name: clone-app-repo
taskRef:
name: git-clone-from-catalog
params:
- name: url
value: https://github.com/tektoncd/community.git
- name: subdirectory
value: application
workspaces:
- name: output
workspace: git-source
finally:
# Cleanup workspace
- name: cleanup
taskRef:
name: cleanup-workspace
workspaces:
- name: source
workspace: git-source
---

# PipelineRun to execute pipeline - clone-into-workspace-and-cleanup-workspace
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: write-and-cleanup-workspace
spec:
pipelineRef:
name: clone-cleanup-workspace
serviceAccountName: 'default'
workspaces:
- name: git-source
volumeClaimTemplate:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
Loading

0 comments on commit 3e22d0e

Please sign in to comment.