pipeline level finally - implementation

We can now specify a list of tasks needs to be executed just before
pipeline exits (either after finishing all non-final tasks successfully or after
a single failure)

Most useful for tasks such as report test results, cleanup cluster resources, etc

```
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: pipeline-with-final-tasks
spec:
  tasks:
    - name: pre-work
      taskRef:
        Name: some-pre-work
    - name: unit-test
      taskRef:
        Name: run-unit-test
      runAfter:
        - pre-work
    - name: integration-test
      taskRef:
        Name: run-integration-test
      runAfter:
        - unit-test
  finally:
    - name: cleanup-test
      taskRef:
        Name: cleanup-cluster
    - name: report-results
      taskRef:
        Name: report-test-results
```

docs/pipelines.md

@@ -21,6 +21,7 @@ weight: 3
   - [Configuring execution results at the `Pipeline` level](#configuring-execution-results-at-the-pipeline-level)
   - [Configuring the `Task` execution order](#configuring-the-task-execution-order)
   - [Adding a description](#adding-a-description)
+  - [Adding `Finally` to the `Pipeline`](#adding-finally-to-the-pipeline)
   - [Code examples](#code-examples)
 
 ## Overview
@@ -516,6 +517,196 @@ In particular:
 
 The `description` field is an optional field and can be used to provide description of the `Pipeline`.
 
+## Adding `Finally` to the `Pipeline`
+
+You can specify a list of one or more final tasks under `finally` section. Final tasks are guaranteed to be executed
+after all `PipelineTasks` under `tasks` have completed regardless of success or error. Final tasks are very similar to
+`PipelineTasks` under `tasks` section and follow the same syntax. Each final task must have a
+[valid](https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names) `name` and a `taskRef` or
+`taskSpec`. For example:
+
+```yaml
+spec:
+  tasks:
+    - name: tests
+      taskRef:
+        Name: integration-test
+  finally:
+    - name: cleanup-test
+      taskRef:
+        Name: cleanup
+```
+
+### Specifying `Workspaces` in Final Tasks
+
+It is possible that the final tasks might require access to the same workspaces which `PipelineTasks` might have utilized
+e.g. a mount point for credentials held in Secrets. To support that requirement, you can specify one or more
+`Workspaces` in the `workspaces` field for the final tasks similar to `tasks`.
+
+```yaml
+spec:
+  resources:
+    - name: app-git
+      type: git
+  workspaces:
+    - name: shared-workspace
+  tasks:
+    - name: clone-app-source
+      taskRef:
+        name: clone-app-repo-to-workspace
+      workspaces:
+        - name: shared-workspace
+          workspace: shared-workspace
+      resources:
+        inputs:
+          - name: app-git
+            resource: app-git
+  finally:
+    - name: cleanup-workspace
+      taskRef:
+        name: cleanup-workspace
+      workspaces:
+        - name: shared-workspace
+          workspace: shared-workspace
+```
+
+### Specifying `Parameters` in Final Tasks
+
+Again, similar to `tasks`, you can specify [`Parameters`](tasks.md#specifying-parameters):
+
+```yaml
+spec:
+  tasks:
+    - name: tests
+      taskRef:
+        Name: integration-test
+  finally:
+    - name: report-results
+      taskRef:
+        Name: report-results
+      params:
+        - name: url
+          value: $(params.url)
+```
+
+### `PipelineRun` Status with `finally`
+
+With `finally`, `PipelineRun` status is calculated based on `PipelineTasks` under `tasks` section and final tasks.
+
+Without `finally`:
+
+| `PipelineTasks` under `tasks` | `PipelineRun` status | Reason |
+| ----------------------------- | -------------------- | ------ |
+| all `PipelineTasks` successful | `true` | `Succeeded` |
+| one or more `PipelineTasks` skipped and rest successful | `true` | `Completed` |
+| single failure of `PipelineTask` | `false` | `failed` |
+
+With `finally`:
+
+| `PipelineTasks` under `tasks` | Final Tasks | `PipelineRun` status | Reason |
+| ----------------------------- | ----------- | -------------------- | ------ |
+| all `PipelineTask` successful | all final tasks successful | `true` | `Succeeded` |
+| all `PipelineTask` successful | one or more failure of final tasks | `false` | `Failed` |
+| one or more `PipelineTask` skipped and rest successful | all final tasks successful | `true` | `Completed` |
+| one or more `PipelineTask` skipped and rest successful | one or more failure of final tasks | `false` | `Failed` |
+| single failure of `PipelineTask` | all final tasks successful | `false` | `failed` |
+| single failure of `PipelineTask` | one or more failure of final tasks | `false` | `failed` |
+
+### Known Limitations
+
+### Specifying `Resources` in Final Tasks
+
+Similar to `tasks`, you can use [PipelineResources](#specifying-resources) as inputs and outputs for
+final tasks in the Pipeline. The only difference here is, final tasks with an input resource can not have a `from` clause
+like a `PipelineTask` from `tasks` section. For example:
+
+```yaml
+spec:
+  tasks:
+    - name: tests
+      taskRef:
+        Name: integration-test
+      resources:
+        inputs:
+          - name: source
+            resource: tektoncd-pipeline-repo
+        outputs:
+          - name: workspace
+            resource: my-repo
+  finally:
+    - name: clear-workspace
+      taskRef:
+        Name: clear-workspace
+      resources:
+        inputs:
+          - name: workspace
+            resource: my-repo
+            from:
+              - tests
+```
+
+### Cannot configure the Final Task execution order
+
+It's not possible to configure or modify the execution order of the final tasks. Unlike `Tasks` in a `Pipeline`,
+all final tasks run simultaneously and starts executing once all `PipelineTasks` under `tasks` have settled which means
+no `runAfter` can be specified in final tasks.
+
+### Cannot specify execution `Conditions` in Final Tasks
+
+`Tasks` in a `Pipeline` can be configured to run only if some conditions are satisfied using `conditions`. But the
+final tasks are guaranteed to be executed after all `PipelineTasks` therefore no `conditions` can be specified in
+final tasks.
+
+#### Cannot configure `Task` execution results with `finally`
+
+Final tasks can not be configured to consume `Results` of `PipelineTask` from `tasks` section i.e. the following
+example is not supported right now but we are working on adding support for the same.
+
+```yaml
+spec:
+  tasks:
+    - name: count-comments-before
+      taskRef:
+        Name: count-comments
+    - name: add-comment
+      taskRef:
+        Name: add-comment
+    - name: count-comments-after
+      taskRef:
+        Name: count-comments
+  finally:
+    - name: check-count
+      taskRef:
+        Name: check-count
+      params:
+        - name: before-count
+          value: $(tasks.count-comments-before.results.count)
+        - name: after-count
+          value: $(tasks.count-comments-after.results.count)
+```
+
+#### Cannot configure `Pipeline` result with `finally`
+
+Final tasks can emit `Results` but results emitted from the final tasks can not be configured in the
+[Pipeline Results](#configuring-execution-results-at-the-pipeline-level). We are working on adding support for this.
+
+```yaml
+  results:
+    - name: comment-count-validate
+      value: $(finally.check-count.results.comment-count-validate)
+```
+
+In this example, `PipelineResults` is set to:
+
+```
+"pipelineResults": [
+  {
+    "name": "comment-count-validate",
+    "value": "$(finally.check-count.results.comment-count-validate)"
+  }
+],
+```
+
 ## Code examples
 
 For a better understanding of `Pipelines`, study [our code examples](https://github.com/tektoncd/pipeline/tree/master/examples).

pkg/apis/pipeline/v1beta1/pipeline_types.go

@@ -71,6 +71,10 @@ type PipelineSpec struct {
 	// Results are values that this pipeline can output once run
 	// +optional
 	Results []PipelineResult `json:"results,omitempty"`
+	// Finally declares the list of Tasks that execute just before leaving the Pipeline
+	// i.e. either after all Tasks are finished executing successfully
+	// or after a failure which would result in ending the Pipeline
+	Finally []PipelineTask `json:"finally,omitempty"`
 }
 
 // PipelineResult used to describe the results of a pipeline

pkg/reconciler/pipelinerun/pipelinerun.go

@@ -394,6 +394,23 @@ func (c *Reconciler) reconcile(ctx context.Context, pr *v1beta1.PipelineRun) err
 		return nil
 	}
 
+	// build DAG with a list of final tasks, this DAG is used later to identify
+	// if a task from PipelineRunState is one of the tasks specified under finally section
+	// the finally section is optional and might not be specified
+	// in case this section is not specified, dfinally holds an empty Graph
+	dfinally, err := dag.Build(v1beta1.PipelineTaskList(pipelineSpec.Finally))
+	if err != nil {
+		// This Run has failed, so we need to mark it as failed and stop reconciling it
+		pr.Status.SetCondition(&apis.Condition{
+			Type:   apis.ConditionSucceeded,
+			Status: corev1.ConditionFalse,
+			Reason: ReasonInvalidGraph,
+			Message: fmt.Sprintf("PipelineRun %s's Pipeline DAG is invalid: %s",
+				fmt.Sprintf("%s/%s", pr.Namespace, pr.Name), err),
+		})
+		return nil
+	}
+
 	if err := pipelineSpec.Validate(ctx); err != nil {
 		// This Run has failed, so we need to mark it as failed and stop reconciling it
 		pr.Status.SetCondition(&apis.Condition{
@@ -472,6 +489,9 @@ func (c *Reconciler) reconcile(ctx context.Context, pr *v1beta1.PipelineRun) err
 	// Apply parameter substitution from the PipelineRun
 	pipelineSpec = resources.ApplyParameters(pipelineSpec, pr)
 
+	// pipelineState holds a list of pipeline tasks with resolved conditions and pipeline resources
+	// pipelineState also holds a taskRun for each pipeline task after the taskRun is created
+	// pipelineState is instantiated on every reconcile cycle
 	pipelineState, err := resources.ResolvePipelineRun(ctx,
 		*pr,
 		func(name string) (v1beta1.TaskInterface, error) {
@@ -486,7 +506,7 @@ func (c *Reconciler) reconcile(ctx context.Context, pr *v1beta1.PipelineRun) err
 		func(name string) (*v1alpha1.Condition, error) {
 			return c.conditionLister.Conditions(pr.Namespace).Get(name)
 		},
-		pipelineSpec.Tasks, providedResources,
+		append(pipelineSpec.Tasks, pipelineSpec.Finally...), providedResources,
 	)
 
 	if err != nil {
@@ -556,12 +576,25 @@ func (c *Reconciler) reconcile(ctx context.Context, pr *v1beta1.PipelineRun) err
 		}
 	}
 
+	// candidateTasks contains a list of pipeline tasks that can be scheduled next
+	// This list of candidates is derived based on the successfully finished tasks
+	// A task is considered as candidate if its all parents have finished executing successfully
+	// candidateTasks are initialized with all root/s of DAG tasks
 	candidateTasks, err := dag.GetSchedulable(d, pipelineState.SuccessfulPipelineTaskNames()...)
 	if err != nil {
 		c.Logger.Errorf("Error getting potential next tasks for valid pipelinerun %s: %v", pr.Name, err)
 	}
 
+	// GetNextTasks returns a list of tasks which should be executed next and is derived based on candidateTasks
+	// GetNextTasks returns a list of candidates for which pipelineState does not have any taskRun or a list of
+	// failed tasks which haven't exhausted their retries
+	// Pipeline execution continues until GetNextTasks does not return any new pipeline task to execute
 	nextRprts := pipelineState.GetNextTasks(candidateTasks)
+	// GetFinalTasks returns a list of final tasks without any taskRun associated with it
+	// GetFinalTasks returns final tasks only when all DAG tasks have finished executing successfully or
+	// executing any one DAG task resulted in failure
+	nextRprts = append(nextRprts, pipelineState.GetFinalTasks(d, dfinally)...)
+
 	resolvedResultRefs, err := resources.ResolveResultRefs(pipelineState, nextRprts)
 	if err != nil {
 		c.Logger.Infof("Failed to resolve all task params for %q with error %v", pr.Name, err)