Proposal: Dynamic Output Resources

[chhsia0]

Expected Behavior

Currently a PipelineRun binds a task's output to predefined static resources, which is not intuitive. A more natural way to model a pipeline is to dynamically generate new resources based on predefined templates as the output of a task. For example:

apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: build-and-push
spec:
  resources:
  - name: src
    type: git
  params:
  - name: imageUrl
    type: string
  steps:
  - name: kaniko
    image: gcr.io/kaniko-project/executor
    command:
    - /kaniko/executor
    args:
    - --context=/workspace/src
    - --destination=$(params.imageUrl)
    - --digest-file=/workspace/digest
    outputs:
    - name: digest
      fromPath: /workspace/digest
  outputs:
    resources:
    - name: img
      spec:
        type: image
        params:
        - name: url
          value: $(params.imageUrl)
        - name: digest
          value: $(steps.kaniko.digest)
---
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: run-test-binary
spec:
  resources:
  - name: img
    type: image
  params:
  - name: cmd
    type: array
  steps:
  - name: run-test
    image: $(resources.img.url)@$(resources.img.digest)
    workingDir: /
    command: ["$(params.cmd)"]
---
apiVersion: tekton.dev/v1alpha1
kind: Pipeline
metadata:
  name: build-and-test
spec:
  resources:
  - name: src
    type: git
  params:
  - name: imageUrl
    type: string
  tasks:
  - name: build-and-push
    taskRef:
      name: build-and-push
    resources:
    - name: src
      resourceRef:
        name: $(resources.src) # Resolves to the resource object name.
    params:
    - name: imageUrl
      value: $(params.imageUrl)
  - name: run-test-binary
    taskRef:
      name: run-test-binary
    resources:
    - name: img
      resourceRef:
        name: $(tasks.build-and-push.img) # This also specifies input dependency.
    params:
    - name: cmd
      value: ['/knative-route-demo.test']
---
apiVersion: tekton.dev/v1alpha1
kind: PipelineResource
metadata:
  name: knative-route-demo-git
spec:
  type: git
  params:
  - name: revision
    value: master
  - name: url
    value: https://github.com/chhsia0/knative-route-demo
---
apiVersion: tekton.dev/v1alpha1
kind: PipelineRun
metadata:
  name: build-and-test-run
spec:
  pipelineRef:
    name: build-and-test
  resources:
  - name: src
    resourceRef:
      name: knative-route-demo-git
  params:
  - name: imageUrl
    value: docker.io/chhsiao/knative-route-demo

In the above example, the PipelineRun and Pipeline does not bind any predefined image resource to the img output resource of Task build-and-push. Instead, Task build-and-push provides how the output image resource should look like, and defines output values for each step that can be used for templating the output image resource. Ideally, the following resource would be generated automatically:

apiVersion: tekton.dev/v1alpha1
kind: PipelineResource
metadata:
  name: build-and-push-image-pbtqf
  labels:
    tekton.dev/task: build-and-push
    tekton.dev/taskrun: build-and-push-run
  ownerReferences:
  - apiVersion: tekton.dev/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: PipelineRun
    name: build-and-test-run
    uid: 4802d6d7-b3eb-11e9-855b-0aed1dbc4ac8
spec:
  type: image
  params:
  - name: url
    value: docker.io/chhsiao/knative-route-demo
  - name: digest
    value: sha256:54eda95dcc205717e35f96a257b6aab16b9817870ede5739ef739484a27b8678

Since output resources are not statically bound, they are immutable once generated. The deletion of the PipelineRun should clean up all output resources it generates.

We could further support generating arbitrary resources like what a TriggerTemplate does:

apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: build-and-deploy
spec:
  ...
  outputs:
    resources:
    - ...
    objects:
    - name: knative-service
      resourceTemplate:
        apiVersion: serving.knative.dev/v1alpha1
        kind: Service
        metadata:
          name: blue-green-demo
        spec:
          template:
            spec:
              containers:
              - image: $(inputs.params.imageUrl)@$(steps.kaniko.digest)

This would immediately deploy a new Knative service YAML, provided that Knative serving is installed in the cluster. Otherwise the TaskRun should fail, with the side effect that the built image is uploaded by the kaniko step. The generated output objects are not PipelineResources, thus cannot be used as inputs to other Tasks in a Pipeline.

Actual Behavior

The current designs of Task, TaskRun and PipelineRun impose static bindings between
pre-existing PipelineResources and the output resources declarations of Tasks, and there is no way to "update" the output resources with data produced in Tasks. As a result, there are
workarounds such as #721, and issues like #216 and #552.

Also, right now deploying an image built by a Task in a Pipeline usually involves defining another Task that 1. fetches a resource YAML from some source repository, 2. modifies the resource YAML through e.g. yq or kustomize to use the built image, and 3. uses kubectl to upload the modified resource YAML. The is fairly routine and should have a built-in mechanism in Tekton pipeline.

The proposed mechanism of dynamic output resources could resolve the above issues.

Additional Info

Output Resource Naming

There are a couple use cases to consider:

1. Static names. This could be achieved as follows:

   apiVersion: tekton.dev/v1alpha1
   kind: Task
   metadata:
     name: build-and-push
   spec:
     ...
     outputs:
       resources:
       - name: image
         metadata:
           name: knative-route-demo-image-v1
         spec:
           ...
   

   If a PipelineResource with the same name already exists, the Task should fail.

2. Dynamic names that are not referenced by other resources. This seems to be the typical use case and the proposed mechanism relies on Kubernetes generating names for the output resources.

3. Dynamically names that need to be referenced by other Tasks. I'm not proposing any solution here, but it might be good to align with Add Support For Naming Modifiers triggers#50.

Implementation

A possible implementation is to run a sidecar container that receives key-value pairs through HTTP POST requests for output resource tempating. At the end of each step, the redirector birany would send the outputs of the step to the sidecar. Once all template variables are resolved, the sidecar could generate the output resources and submit them to the API server; it could serve the resource YAML through HTTP GET requests, and use another step to control when to submit the output resources.

[bobcatfish]

I've added this as a proposal to solve #1279 (https://docs.google.com/document/d/13TZdWU6PEaOwpe--Zw96lcpjRqksJ7zAQ-mxk1qnZ30/edit#).

[chhsia0]

Updated the example to use Pipeline and PipelineRun. Also updated the above design doc.

[bobcatfish]

I'm tempted to close this for now in light of our complete revisiting of PipelineResources in #1673, plz reopen if you disagree @chhsia0 !