[TEP 0075, 0076]: Structured object / arrays in Results and Params

[ywluogg]

Feature request

The issue will track the implementation for TEP-0075 Propose object (dicitonary) param and result types and TEP-0076 Propose array results and indexing

The two docs are in the process of review to get to Implementable states (tektoncd/community#661)

Use case

Will have a more detailed use case summary, but please refer to the use cases mentioned in TEP-0075 Propose object (dicitonary) param and result types and TEP-0076 Propose array results and indexing

To summarize, we want to change the params to be supporting the following:

Params will affect the following:

• Pipeline.params
• Pipeline.TaskRef.params
• PipelineRun.params
• PipelineRun.PipelineSpec.params
• PipelineRun.PipelineSpec.tasks.params
• PipelineRun.PipelineSpec.taskSpec.params
• Task.params
• TaskRun.spec.params
• TaskRun.TaskSpec,params

We also want to change the results to be supporting objects and arrays in Pipeline Results and Task Results.
Results will affect the following:

• PipelineRun.Spec.PipelineSpec.Results
• PipelineRun.Spec.PipelineSpec.PipelineSpec.TaskSpec.Results
• Pipeline.Spec.Results
• TaskRun.Spec.TaskSpec.Results
• Task.Spec.Results

Below an example to specify a object param in Tasks and using its key1 and image1 value, and also specify a object in Task results and use script to write the results:

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: task-with-parameters
spec:
  params:
  - name: objectVal1
     description: "description1"
     type: object
     properties:
        key1:
          type: string
        image1:
          type: string
  - name: objectVal2
     description: "description2"
     type: object
     properties:
        key1:
          type: string
        image1:
          type: string
  steps:
  - name: step1
     image: $(params.objectVal1.image1)
     script: |
     echo $(params.objectVal1.key1)
  - name: step2
     image: $(params.objectVal2.image1)
     script: |
     echo $(params.objectVal2.key1)
  - name: step3
     image: bash:latest
     script: |
          #!/usr/bin/env bash
          echo -n "{"key1": "$(params.objectVal1.key1)", "image1": "$(params.objectVal1.image1)"}" | tee $(results.resultObject1.path)
  results:
  - name: resultObject1
    description: "description3"
     type: object
     properties:
        key1:
          type: string
        image1:
          type: string

Below an example to specify a object param in TaskRuns:

apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: taskrun1
spec:
  params:
  - name: objectVal1
     value:
        key1: value1
        image1: bash:latest
  - name: objectVal2
     value:
        key1: value2
        image1: bash:latest
  taskRef:
    name: task-with-parameters

Below an example to specify a object param in PipelineRun and use it as a whole in tasks:

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: pipelinerun-object-param
spec:
  pipelineSpec:
    params:
      - name: pipelineObjectVal1
        description: "description1"
        type: object
        properties:
            key1:
              type: string
            image1:
              type: string
        default:
            key1: "val1"
            image1: "bash:latest"
    tasks:
      - name: task-1
        taskRef:
          name: task-with-parameters
        params:
          - name: objectVal1
             value: $(params.pipelineObjectVal1[*])
          - name: objectVal2
             value: $(params.pipelineObjectVal1[*])
      - name: task-2
        taskRef:
          name: task-with-parameters
        params:
          - name: objectVal1
             value: $(tasks.task-1.params.resultObject1[*])
          - name: objectVal2
             value: $(tasks.task-1.params.resultObject1[*])
  params:
    - name: pipelineObjectVal1
       value:
          key1: "val1"
          image1: "bash:latest"
     

Below an example to specify a object param in a Task inPipelineRun and use it as a whole in:

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: pipelinerun-object-param
spec:
  pipelineSpec:
    params:
      - name: pipelineObjectVal1
        description: "description1"
        type: object
        properties:
            key1:
              type: string
            image1:
              type: string
        default:
            key1: val1
            image1: bash:latest
    tasks:
      - name: task-1
        taskRef:
          name: task-with-parameters
        params:
          - name: objectVal1
             value: $(params.pipelineObjectVal1[*])
          - name: objectVal2
             value: $(params.pipelineObjectVal1[*])
    results:
    - name: pipelineResult1
       description: "result description 1"
       value: $(tasks.task-1.results.resultObject1[*])
    - name: pipelineResult2
       description: "result description 2"
       value:
          key2: $(tasks.task-1.results.resultObject1.key1)
  params:
    - name: pipelineObjectVal1
      value:
         key1: val2
         image1: bash:latest

General Implementation Requirements

• First Stage: implement features in v1beta1 apiversion and hide them under alpha feature flag,

Task Breakdowns

• Area 1
  • Object in Params
    • Task, TaskRun webhook validations on object params
    • Task, TaskRun webhook mutations on object params
    • Pipeline, PipelineRun webhook validations on object params
    • Object vs object properties validations
    • TaskRun & PipelineRun reconciler substitutions on object params: individual keys and as a whole
    • Param value type corrections
    • TaskRun against Task spec & PipelineRun against Pipeline spec reconciler validations on object params using object vs object properties validations
    • E2E Tests for object params
    • Example yamls for object params in TaskRuns and PipelineRuns
• Area 2
  • Array types
    • Support array type of results spec in Task and Pipeline
    • Support array type of results for emitting in file and consuming from file
    • Task webhook validations on array results
    • Pipeline webhook validations on array results
    • Param and Result array indexing substitutions
    • E2E Tests for results in TaskRuns and PipelineRuns on array params
    • Example yamls for results in TaskRuns and PipelineRuns on arrays
  • Object type of Results
    • Task and Pipeline webhook validations on object results
    • Results emissions and consuming from files on object results
    • TaskRun & PipelineRun reconciler substitutions on object results: individual keys and as a whole
    • TaskRun against Task spec & PipelineRun against Pipeline spec reconciler validations on object results using object vs object properties validations
  • E2E Tests for results in TaskRuns and PipelineRuns on objects
  • Example yamls for results in TaskRuns and PipelineRuns on objects

Future Work and Discussions

• Renaming of ArrayOrString
• Matrix support objects and params (non-goal for TEP 75 and 76)
• Nested objects (non-goal for TEP 75 and 76)
• Support more ways of object param & results of substitutions ([TEP 0075, 0076] More ways of referencing params and results #5056) (non-goal for TEP 75 and 76)
• Refactoring params and results into separate directories (Refactor results and params into separate sub directories #5001 )
• CRD for objects (non-goal for TEP 75 and 76)
• Nested arrays of objects (non-goal for TEP 75 and 76)
• Allowing usage of object / array as a whole in format results.result1 and params.param1 (Discussion about disallowing usage of object/array param/result as a whole in format: results.result1 or params.param1 (without star reference) #5078)

[dibyom]

/assign @Yongxuanzhang

[tekton-robot]

@dibyom: GitHub didn't allow me to assign the following users: Yongxuanzhang.

Note that only tektoncd members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @Yongxuanzhang

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Yongxuanzhang]

/assign @Yongxuanzhang
I'm working on Area2 and Chuang's working on Area1

[chuangw6]

/assign

[chuangw6]

Hi @ywluogg ,
I just realized that we might need to disallow the parameters whose name contains dot(.). Reasons are mentioned in #4797 , which is related to TEP-0075.

Do you have any thoughts on this?
Thanks!

[ywluogg]

Hi @ywluogg , I just realized that we might need to disallow the parameters whose name contains dot(.). Reasons are mentioned in #4797 , which is related to TEP-0075.

Do you have any thoughts on this? Thanks!

Thanks Chuang for the updates!

[Yongxuanzhang]

Do we miss array indexing for area 2 project? And I think it should be after PipelineResults and PipelineRunResults support for arrays and objects

[Yongxuanzhang]

It may be worth discussing the support for custom tasks when we finish these two teps

[ywluogg]

It may be worth discussing the support for custom tasks when we finish these two teps

+1 I will organize a list of extended items for the two TEPs

[ywluogg]

PRs that need to be added to milestone: #5139 #5132 #5088

[ywluogg]

All the features listed not in "Future Work" section are finished

[ywluogg]

...
status:
  ...
  taskRunResults:
      - name: IMAGE_URL
         value: gcr.io/foo/bar
      - name: IMAGE_DIGEST
         value: sha256:05f95b26ed10668b7183c1e2da98610e91372fa9f510046d4ce5812addad86b7
...

status:
  ...
  taskRunResults:
      - name: image1
         value:
             uri: gcr.io/foo/bar
             digest: sha256:05f95b26ed10668b7183c1e2da98610e91372fa9f510046d4ce5812addad86b7
       - name: ARTIFACT_OUTPUTS
         value:
             - image1
             - maven_pkg
 ...