Passing parameters and resource Pipeline -> Task

[afrittoli]

Expected Behavior

Writing a pipeline does not add a lot of overhead to the single Tasks

Actual Behavior

Today writing a pipeline that includes many parameters and resources results in a rather verbose YAML. It has the advantage that is very explicit - but to the point of being redundant.
I think a thinner syntax on the parameters and resource passing would make the process of writing pipelines simpler.

The proposal is to implicitly pass pipeline parameters and resources with a matching name into the task parameter and resources. So instead of having:

apiVersion: tekton.dev/v1alpha1
kind: Pipeline
metadata:
  name: verify-deploy-test-tekton-release
spec:
  params:
  - name: projectName
    description: Name of the Tekton project to install e.g. pipeline, trigger, dashboard, experimental
    default: pipeline
  - name: package
    description: Name of the Tekton package
    default: github.com/tektoncd/pipeline
  - name: version
    description: The vX.Y.Z version that we want to install (including `v`)
  - name: namespace
    description: Namespace where the Tekton project is installed by the release
  - name: resources
    description: space separated list of resources to be deleted
  - name: container-registry
    description: Container registry where to upload images during tests
  - name: default-service-account
    description: Service account to be setup as default in Tekton configmap
  resources:
  - name: bucket
    type: storage
  - name: test-cluster
    type: cluster
  - name: plumbing
    type: git
  - name: tests
    type: git
  - name: results-bucket
    type: storage
  tasks:
    - name: verify
      taskRef:
        name: verify-tekton-release-github
      params:
        - name: projectName
          value: $(params.projectName)
        - name: version
          value: $(params.version)
      resources:
        inputs:
          - name: release-bucket
            resource: bucket
    - name: deploy
      runAfter: [verify]
      taskRef:
        name: install-tekton-release
      params:
        - name: projectName
          value: $(params.projectName)
        - name: version
          value: $(params.version)
        - name: namespace
          value: $(params.namespace)
      resources:
        inputs:
          - name: release-bucket
            resource: bucket
          - name: k8s-cluster
            resource: test-cluster
          - name: plumbing-library
            resource: plumbing
    - name: log
      runAfter: [deploy]
      taskRef:
        name: log-test-image-tools
      resources:
        inputs:
          - name: plumbing-library
            resource: plumbing
    - name: e2e-test
      runAfter: [log]
      taskRef:
        name: e2e-tests
      params:
      - name: package
        value: $(params.package)
      - name: container-registry
        value: $(params.container-registry)
      resources:
        inputs:
          - name: plumbing-library
            resource: plumbing
          - name: tests
            resource: tests
          - name: test-cluster
            resource: test-cluster
        outputs:
          - name: results-bucket
            resource: results-bucket
    - name: cleanup
      runAfter: [e2e-test]
      taskRef:
        name: cleanup-tekton-release
      params:
        - name: projectName
          value: $(params.projectName)
        - name: namespace
          value: $(params.namespace)
        - name: resources
          value: $(params.resources)
        - name: version
          value: $(params.version)
      resources:
        inputs:
          - name: plumbing-library
            resource: plumbing
          - name: k8s-cluster
            resource: test-cluster
    - name: deploy2
      runAfter: [cleanup]
      taskRef:
        name: install-tekton-release
      params:
        - name: projectName
          value: $(params.projectName)
        - name: version
          value: $(params.version)
        - name: namespace
          value: $(params.namespace)
      resources:
        inputs:
          - name: release-bucket
            resource: bucket
          - name: k8s-cluster
            resource: test-cluster
          - name: plumbing-library
            resource: plumbing
    - name: log2
      runAfter: [deploy2]
      taskRef:
        name: log-test-image-tools
      resources:
        inputs:
          - name: plumbing-library
            resource: plumbing
    - name: e2e-yaml-test
      runAfter: [log2]
      taskRef:
        name: e2e-yaml-tests
      params:
      - name: container-registry
        value: $(params.container-registry)
      resources:
        inputs:
          - name: plumbing-library
            resource: plumbing
          - name: tests
            resource: tests
          - name: test-cluster
            resource: test-cluster
        outputs:
          - name: results-bucket
            resource: results-bucket
    - name: cleanup2
      runAfter: [e2e-yaml-test]
      taskRef:
        name: cleanup-tekton-release
      params:
        - name: projectName
          value: $(params.projectName)
        - name: namespace
          value: $(params.namespace)
        - name: resources
          value: $(params.resources)
        - name: version
          value: $(params.version)
      resources:
        inputs:
          - name: plumbing-library
            resource: plumbing
          - name: k8s-cluster
            resource: test-cluster
    - name: test-results
      runAfter: [cleanup2]
      taskRef:
        name: test-results
      resources:
        inputs:
          - name: results-bucket
            resource: results-bucket

We would be able to write:

apiVersion: tekton.dev/v1alpha1
kind: Pipeline
metadata:
  name: verify-deploy-test-tekton-release
spec:
  params:
  - name: projectName
    description: Name of the Tekton project to install e.g. pipeline, trigger, dashboard, experimental
    default: pipeline
  - name: package
    description: Name of the Tekton package
    default: github.com/tektoncd/pipeline
  - name: version
    description: The vX.Y.Z version that we want to install (including `v`)
  - name: namespace
    description: Namespace where the Tekton project is installed by the release
  - name: resources
    description: space separated list of resources to be deleted
  - name: container-registry
    description: Container registry where to upload images during tests
  - name: default-service-account
    description: Service account to be setup as default in Tekton configmap
  resources:
  - name: release-bucket
    type: storage
  - name: k8s-cluster
    type: cluster
  - name: plumbing-library
    type: git
  - name: tests
    type: git
  - name: results-bucket
    type: storage
  tasks:
    - name: verify
      taskRef:
        name: verify-tekton-release-github
    - name: deploy
      runAfter: [verify]
      taskRef:
        name: install-tekton-release
    - name: log
      runAfter: [deploy]
      taskRef:
        name: log-test-image-tools
    - name: e2e-test
      runAfter: [log]
      taskRef:
        name: e2e-tests
    - name: cleanup
      runAfter: [e2e-test]
      taskRef:
        name: cleanup-tekton-release
    - name: deploy2
      runAfter: [cleanup]
      taskRef:
        name: install-tekton-release
    - name: log2
      runAfter: [deploy2]
      taskRef:
        name: log-test-image-tools
    - name: e2e-yaml-test
      runAfter: [log2]
      taskRef:
        name: e2e-yaml-tests
    - name: cleanup2
      runAfter: [e2e-yaml-test]
      taskRef:
        name: cleanup-tekton-release
    - name: test-results
      runAfter: [cleanup2]
      taskRef:
        name: test-results

Since task names are not standardized, it may be that a pipeline has to pass the same parameter to two or more different tasks where the parameter has a different name. To cover this case we could add an extra field to the pipeline parameter to list aliases.
This approach has the main disadvantage of being more implicit, but it still has the advantage of highlighting which params / resources are not provided by the pipeline - if any.

[vdemeester]

I tend to like that idea.

Since task names are not standardized, it may be that a pipeline has to pass the same parameter to two or more different tasks where the parameter has a different name. To cover this case we could add an extra field to the pipeline parameter to list aliases.

Or we could let it like this, aka if the name do not match, be explicit about it.

The only disadvantage I can see, might be with really common parameter names. Something like package could be used differently in 2 different task, and thus the implicit behaviour might be the wrong one. The other question is, if I'm using a pipeline parameter expelicitely, with a different name, should we still pass it as implicit or as it's already been used, should we skip it ? This one can be a bit tricky as we may be using the parameter in a composition way.

    - name: deploy2
      runAfter: [cleanup]
      taskRef:
        name: install-tekton-release
      params:
        - name: projectName
          value: my-project-$(params.projectName)
        - name: package
          value: $(params.namespace)/$(params.projectName)

/kind feature

[afrittoli]

@vdemeester Good point - and probably implicit is better than explicit. This might be something to be solved at the UI level (Web UI or CLI) - i.e. the UI could offer default matching or parameters and resources at authoring time, and let the author fix things manually were needed.

I think I'll keep this one in the backlog, it may be something to consider when we get to Pipeline authoring tools.

[bobcatfish]

wow this is a cool idea - i actually really like it, it reminds me of a lot of what i used to like about c# development and the associated tools; at least in the shop i was working in, tools could help you out an awful lot b/c all the class and variable naming was super standardized

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[afrittoli]

/remove-lifecycle stale
/remove-lifecycle rotten
/reopen

[tekton-robot]

@afrittoli: Reopened this issue.

In response to this:

/remove-lifecycle stale
/remove-lifecycle rotten
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[vdemeester]

/remove-lifecycle stale

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jerop]

/reopen
/lifecycle frozen

yet to be addressed

[tekton-robot]

@jerop: Reopened this issue.

In response to this:

/reopen
/lifecycle frozen

yet to be addressed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.