implementing pipeline level finally

docs/pipelines.md

@@ -21,7 +21,7 @@ weight: 3
   - [Configuring execution results at the `Pipeline` level](#configuring-execution-results-at-the-pipeline-level)
   - [Configuring the `Task` execution order](#configuring-the-task-execution-order)
   - [Adding a description](#adding-a-description)
-  - [Adding `Finally` to the `Pipeline` (Preview)](#adding-finally-to-the-pipeline-preview)
+  - [Adding `Finally` to the `Pipeline`](#adding-finally-to-the-pipeline)
   - [Code examples](#code-examples)
 
 ## Overview
@@ -530,10 +530,7 @@ In particular:
 
 The `description` field is an optional field and can be used to provide description of the `Pipeline`.
 
-## Adding `Finally` to the `Pipeline` (Preview)
-
-_Finally type is available in the `Pipeline` but functionality is in progress. Final tasks are can be specified and
-are validated but not executed yet._
+## Adding `Finally` to the `Pipeline`
 
 You can specify a list of one or more final tasks under `finally` section. Final tasks are guaranteed to be executed
 in parallel after all `PipelineTasks` under `tasks` have completed regardless of success or error. Final tasks are very
@@ -553,10 +550,186 @@ spec:
         Name: cleanup
 ```
 
-_[PR #2661](https://github.com/tektoncd/pipeline/pull/2661) is implementing this new functionality by adding support to enable
-final tasks along with workspaces and parameters. `PipelineRun` status is being updated to include execution status of
-final tasks i.e. `PipelineRun` status is set to success or failure depending on execution of `PipelineTasks`, this status
-remains same when all final tasks finishes successfully but is set to failure if any of the final tasks fail._
+### Specifying `Workspaces` in Final Tasks
+
+Finally tasks can specify [workspaces](workspaces.md) which `PipelineTasks` might have utilized
+e.g. a mount point for credentials held in Secrets. To support that requirement, you can specify one or more
+`Workspaces` in the `workspaces` field for the final tasks similar to `tasks`.
+
+```yaml
+spec:
+  resources:
+    - name: app-git
+      type: git
+  workspaces:
+    - name: shared-workspace
+  tasks:
+    - name: clone-app-source
+      taskRef:
+        name: clone-app-repo-to-workspace
+      workspaces:
+        - name: shared-workspace
+          workspace: shared-workspace
+      resources:
+        inputs:
+          - name: app-git
+            resource: app-git
+  finally:
+    - name: cleanup-workspace
+      taskRef:
+        name: cleanup-workspace
+      workspaces:
+        - name: shared-workspace
+          workspace: shared-workspace
+```
+
+### Specifying `Parameters` in Final Tasks
+
+Similar to `tasks`, you can specify [`Parameters`](tasks.md#specifying-parameters) in final tasks:
+
+```yaml
+spec:
+  tasks:
+    - name: tests
+      taskRef:
+        Name: integration-test
+  finally:
+    - name: report-results
+      taskRef:
+        Name: report-results
+      params:
+        - name: url
+          value: "someURL"
+```
+
+### `PipelineRun` Status with `finally`
+
+With `finally`, `PipelineRun` status is calculated based on `PipelineTasks` under `tasks` section and final tasks.
+
+Without `finally`:
+
+| `PipelineTasks` under `tasks` | `PipelineRun` status | Reason |
+| ----------------------------- | -------------------- | ------ |
+| all `PipelineTasks` successful | `true` | `Succeeded` |
+| one or more `PipelineTasks` [skipped](conditions.md) and rest successful | `true` | `Completed` |
+| single failure of `PipelineTask` | `false` | `failed` |
+
+With `finally`:
+
+| `PipelineTasks` under `tasks` | Final Tasks | `PipelineRun` status | Reason |
+| ----------------------------- | ----------- | -------------------- | ------ |
+| all `PipelineTask` successful | all final tasks successful | `true` | `Succeeded` |
+| all `PipelineTask` successful | one or more failure of final tasks | `false` | `Failed` |
+| one or more `PipelineTask` [skipped](conditions.md) and rest successful | all final tasks successful | `true` | `Completed` |
+| one or more `PipelineTask` [skipped](conditions.md) and rest successful | one or more failure of final tasks | `false` | `Failed` |
+| single failure of `PipelineTask` | all final tasks successful | `false` | `failed` |
+| single failure of `PipelineTask` | one or more failure of final tasks | `false` | `failed` |
+
+Overall, `PipelineRun` state transitioning is explained below for respective scenarios:
+
+* All `PipelineTask` and final tasks are successful: `Started` -> `Running` -> `Succeeded`
+* At least one `PipelineTask` skipped and rest successful:  `Started` -> `Running` -> `Completed`
+* One `PipelineTask` failed / one or more final tasks failed: `Started` -> `Running` -> `Failed`
+
+Please refer to the [table](pipelineruns.md#monitoring-execution-status) under Monitoring Execution Status to learn about
+what kind of events are triggered based on the `Pipelinerun` status.
+
+### Known Limitations
+
+### Specifying `Resources` in Final Tasks
+
+Similar to `tasks`, you can use [PipelineResources](#specifying-resources) as inputs and outputs for
+final tasks in the Pipeline. The only difference here is, final tasks with an input resource can not have a `from` clause
+like a `PipelineTask` from `tasks` section. For example:
+
+```yaml
+spec:
+  tasks:
+    - name: tests
+      taskRef:
+        Name: integration-test
+      resources:
+        inputs:
+          - name: source
+            resource: tektoncd-pipeline-repo
+        outputs:
+          - name: workspace
+            resource: my-repo
+  finally:
+    - name: clear-workspace
+      taskRef:
+        Name: clear-workspace
+      resources:
+        inputs:
+          - name: workspace
+            resource: my-repo
+            from: #invalid
+              - tests
+```
+
+### Cannot configure the Final Task execution order
+
+It's not possible to configure or modify the execution order of the final tasks. Unlike `Tasks` in a `Pipeline`,
+all final tasks run simultaneously and start executing once all `PipelineTasks` under `tasks` have settled which means
+no `runAfter` can be specified in final tasks.
+
+### Cannot specify execution `Conditions` in Final Tasks
+
+`Tasks` in a `Pipeline` can be configured to run only if some conditions are satisfied using `conditions`. But the
+final tasks are guaranteed to be executed after all `PipelineTasks` therefore no `conditions` can be specified in
+final tasks.
+
+#### Cannot configure `Task` execution results with `finally`
+
+Final tasks can not be configured to consume `Results` of `PipelineTask` from `tasks` section i.e. the following
+example is not supported right now but we are working on adding support for the same (tracked in issue
+[#2557](https://github.com/tektoncd/pipeline/issues/2557)).
+
+```yaml
+spec:
+  tasks:
+    - name: count-comments-before
+      taskRef:
+        Name: count-comments
+    - name: add-comment
+      taskRef:
+        Name: add-comment
+    - name: count-comments-after
+      taskRef:
+        Name: count-comments
+  finally:
+    - name: check-count
+      taskRef:
+        Name: check-count
+      params:
+        - name: before-count
+          value: $(tasks.count-comments-before.results.count) #invalid
+        - name: after-count
+          value: $(tasks.count-comments-after.results.count) #invalid
+```
+
+#### Cannot configure `Pipeline` result with `finally`
+
+Final tasks can emit `Results` but results emitted from the final tasks can not be configured in the
+[Pipeline Results](#configuring-execution-results-at-the-pipeline-level). We are working on adding support for this
+(tracked in issue [#2710](https://github.com/tektoncd/pipeline/issues/2710)).
+
+```yaml
+  results:
+    - name: comment-count-validate
+      value: $(finally.check-count.results.comment-count-validate)
+```
+
+In this example, `PipelineResults` is set to:
+
+```
+"pipelineResults": [
+  {
+    "name": "comment-count-validate",
+    "value": "$(finally.check-count.results.comment-count-validate)"
+  }
+],
+```
 
 ## Code examples
 

examples/v1beta1/pipelineruns/pipelinerun-with-final-tasks.yaml

@@ -0,0 +1,187 @@
+# Copied from https://github.com/tektoncd/catalog/blob/v1beta1/git/git-clone.yaml :(
+# This can be deleted after we add support to refer to the remote Task in a registry (Issue #1839) or
+# add support for referencing task in git directly (issue #2298)
+apiVersion: tekton.dev/v1beta1
+kind: Task
+metadata:
+  name: git-clone-from-catalog
+spec:
+  workspaces:
+    - name: output
+      description: The git repo will be cloned onto the volume backing this workspace
+  params:
+    - name: url
+      description: git url to clone
+      type: string
+    - name: revision
+      description: git revision to checkout (branch, tag, sha, ref…)
+      type: string
+      default: master
+    - name: refspec
+      description: (optional) git refspec to fetch before checking out revision
+      default: ""
+    - name: submodules
+      description: defines if the resource should initialize and fetch the submodules
+      type: string
+      default: "true"
+    - name: depth
+      description: performs a shallow clone where only the most recent commit(s) will be fetched
+      type: string
+      default: "1"
+    - name: sslVerify
+      description: defines if http.sslVerify should be set to true or false in the global git config
+      type: string
+      default: "true"
+    - name: subdirectory
+      description: subdirectory inside the "output" workspace to clone the git repo into
+      type: string
+      default: ""
+    - name: deleteExisting
+      description: clean out the contents of the repo's destination directory (if it already exists) before trying to clone the repo there
+      type: string
+      default: "false"
+    - name: httpProxy
+      description: git HTTP proxy server for non-SSL requests
+      type: string
+      default: ""
+    - name: httpsProxy
+      description: git HTTPS proxy server for SSL requests
+      type: string
+      default: ""
+    - name: noProxy
+      description: git no proxy - opt out of proxying HTTP/HTTPS requests
+      type: string
+      default: ""
+  results:
+    - name: commit
+      description: The precise commit SHA that was fetched by this Task
+  steps:
+    - name: clone
+      image: gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init:v0.12.1
+      script: |
+        CHECKOUT_DIR="$(workspaces.output.path)/$(params.subdirectory)"
+
+        cleandir() {
+          # Delete any existing contents of the repo directory if it exists.
+          #
+          # We don't just "rm -rf $CHECKOUT_DIR" because $CHECKOUT_DIR might be "/"
+          # or the root of a mounted volume.
+          if [[ -d "$CHECKOUT_DIR" ]] ; then
+            # Delete non-hidden files and directories
+            rm -rf "$CHECKOUT_DIR"/*
+            # Delete files and directories starting with . but excluding ..
+            rm -rf "$CHECKOUT_DIR"/.[!.]*
+            # Delete files and directories starting with .. plus any other character
+            rm -rf "$CHECKOUT_DIR"/..?*
+          fi
+        }
+
+        if [[ "$(params.deleteExisting)" == "true" ]] ; then
+          cleandir
+        fi
+
+        test -z "$(params.httpProxy)" || export HTTP_PROXY=$(params.httpProxy)
+        test -z "$(params.httpsProxy)" || export HTTPS_PROXY=$(params.httpsProxy)
+        test -z "$(params.noProxy)" || export NO_PROXY=$(params.noProxy)
+
+        /ko-app/git-init \
+          -url "$(params.url)" \
+          -revision "$(params.revision)" \
+          -refspec "$(params.refspec)" \
+          -path "$CHECKOUT_DIR" \
+          -sslVerify="$(params.sslVerify)" \
+          -submodules="$(params.submodules)" \
+          -depth "$(params.depth)"
+        cd "$CHECKOUT_DIR"
+        RESULT_SHA="$(git rev-parse HEAD | tr -d '\n')"
+        EXIT_CODE="$?"
+        if [ "$EXIT_CODE" != 0 ]
+        then
+          exit $EXIT_CODE
+        fi
+        # Make sure we don't add a trailing newline to the result!
+        echo -n "$RESULT_SHA" > $(results.commit.path)
+
+---
+
+# Task to cleanup shared workspace
+apiVersion: tekton.dev/v1beta1
+kind: Task
+metadata:
+  name: cleanup-workspace
+spec:
+  workspaces:
+    # Shared workspace where git repo is cloned
+    - name: source
+  steps:
+    - name: check-application-dir-has-source
+      image: ubuntu
+      script: |
+        if [ ! -d "$(workspaces.source.path)/application/" ]; then
+          echo "Something went wrong and could not find application source under $(workspaces.source.path)/application/"
+          exit 1
+        fi
+    - name: cleanup-workspace
+      image: ubuntu
+      script: |
+        rm -rf $(workspaces.source.path)/application/
+    - name: verify-application-dir-has-gone
+      image: ubuntu
+      script: |
+        if [ -d "$(workspaces.source.path)/application/" ]; then
+          echo "Something went wrong cleaning up and the application source still exists under $(workspaces.source.path)/application/"
+          exit 1
+        fi
+---
+
+# Pipeline to clone repo into shared workspace and cleanup the workspace after done
+apiVersion: tekton.dev/v1beta1
+kind: Pipeline
+metadata:
+  name: clone-cleanup-workspace
+spec:
+  workspaces:
+    # common workspace where git repo is cloned and needs to be cleanup after done
+    - name: git-source
+  tasks:
+    # Clone app repo to workspace
+    - name: clone-app-repo
+      taskRef:
+        name: git-clone-from-catalog
+      params:
+        - name: url
+          value: https://github.com/tektoncd/community.git
+        - name: subdirectory
+          value: application
+      workspaces:
+        - name: output
+          workspace: git-source
+  finally:
+    # Cleanup workspace
+    - name: cleanup
+      taskRef:
+        name: cleanup-workspace
+      workspaces:
+        - name: source
+          workspace: git-source
+---
+
+# PipelineRun to execute pipeline - clone-into-workspace-and-cleanup-workspace
+apiVersion: tekton.dev/v1beta1
+kind: PipelineRun
+metadata:
+  name: write-and-cleanup-workspace
+spec:
+  pipelineRef:
+    name: clone-cleanup-workspace
+  serviceAccountName: 'default'
+  workspaces:
+    - name: git-source
+      volumeClaimTemplate:
+        spec:
+          accessModes:
+            - ReadWriteOnce
+          resources:
+            requests:
+              storage: 1Gi
+---