v0.17.0 Pipeline can't be Run; it has an invalid spec: missing field(s): tasks[1].when[0]

[r0bj]

Expected Behavior

Pipeline is executed.

Actual Behavior

PipelineValidationFailed:

  Warning  Failed         6m50s (x2 over 6m50s)  PipelineRun  Pipeline tekton-execution/build-images can't be Run; it has an invalid spec: missing field(s): tasks[1].when[0], tasks[2].when[0], tasks[4].when[0]
  Warning  InternalError  6m50s (x2 over 6m50s)  PipelineRun  1 error occurred:
           * missing field(s): tasks[1].when[0], tasks[2].when[0], tasks[4].when[0]

Status:
  Completion Time:  2020-10-13T23:30:43Z
  Conditions:
    Last Transition Time:  2020-10-13T23:30:43Z
    Message:               Pipeline tekton-execution/build-images can't be Run; it has an invalid spec: missing field(s): tasks[1].when[0], tasks[2].when[0], tasks[4].when[0]
    Reason:                PipelineValidationFailed
    Status:                False
    Type:                  Succeeded
  Pipeline Spec:
    Params:
      Name:  git-repo-url
      Type:  string
      Name:  git-revision
      Type:  string
      Name:  git-ref
      Type:  string
    Tasks:
      Name:  validate-git-ref
      Params:
        Name:   git-ref
        Value:  $(params.git-ref)
      Task Ref:
        Kind:  Task
        Name:  validate-git-ref
      Name:    fetch-repository
      Params:
        Name:   url
        Value:  $(params.git-repo-url)
        Name:   revision
        Value:  $(params.git-revision)
        Name:   deleteExisting
        Value:  true
      Task Ref:
        Kind:  Task
        Name:  git-clone
      When:
        Input:
        Operator:
        Values:    <nil>
      Workspaces:
        Name:       output
        Workspace:  build-images-workspace

It seems that "When" fields are empty.

Steps to Reproduce the Problem

PipelineRun definition:

---
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: build-images
  namespace: tekton-execution
spec:
  params:
  - name: git-repo-url
  - name: git-revision
  - name: git-ref
  workspaces:
  - name: build-images-workspace
  tasks:

  - name: validate-git-ref
    taskRef:
      name: validate-git-ref
    params:
    - name: git-ref
      value: $(params.git-ref)

  - name: fetch-repository
    when:
    - input: "$(tasks.validate-git-ref.results.status)"
      operator: in
      values:
      - "true"
    taskRef:
      name: git-clone
    workspaces:
    - name: output
      workspace: build-images-workspace
    params:
    - name: url
      value: $(params.git-repo-url)
    - name: revision
      value: $(params.git-revision)
    - name: deleteExisting
      value: "true"

Additional Info

• Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-14T11:09:22Z", GoVersion:"go1.14.7", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.4", GitCommit:"c96aede7b5205121079932896c4ad89bb93260af", GitTreeState:"clean", BuildDate:"2020-06-17T11:33:59Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

• Tekton Pipeline version:

Client version: 0.13.0
Pipeline version: v0.17.0
Triggers version: v0.8.1

In tekton pipelines 0.16 it works as expected.

[pritidesai]

@jerop please help troubleshoot

[pritidesai]

@r0bj can you please try running an example from the pipeline repo?

https://github.com/tektoncd/pipeline/blob/master/examples/v1beta1/pipelineruns/pipelinerun-with-when-expressions.yaml

[r0bj]

@pritidesai after some more tests it turned out that this issue manifests itself in scenario like this:

1. install tekton pipeline v0.16.3
2. install pipeline definition (CRD)
3. update tekton pipeline to version v0.17.0
4. run pipelinerun

So I needed to reapply all my pipelines definitions in order to make them work.

[pritidesai]

oh 😞, so pipelines deployed in Tekton v0.16.3 not compatible with v0.17.0 and results in such validation error?

[r0bj]

I've prepared yaml definitions based on https://github.com/tektoncd/pipeline/blob/master/examples/v1beta1/pipelineruns/pipelinerun-with-when-expressions.yaml (separate Pipeline and PipelineRun) to be able to reproduce this issue:

1. install tekton pipeline v0.16.3
2. install Pipeline guarded-pr

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: guarded-pr
spec:
  params:
    - name: path
      type: string
      description: The path of the file to be created
  workspaces:
    - name: source
      description: |
        This workspace is shared among all the pipeline tasks to read/write common resources
  tasks:
    - name: create-file # when expression using parameter, evaluates to true
      when:
        - input: "$(params.path)"
          operator: in
          values: ["README.md"]
      workspaces:
        - name: source
          workspace: source
      taskSpec:
        workspaces:
          - name: source
            description: The workspace to create the readme file in
        steps:
          - name: write-new-stuff
            image: ubuntu
            script: 'touch $(workspaces.source.path)/README.md'
    - name: check-file
      params:
        - name: path
          value: "$(params.path)"
      workspaces:
        - name: source
          workspace: source
      runAfter:
        - create-file
      taskSpec:
        params:
          - name: path
        workspaces:
          - name: source
            description: The workspace to check for the file
        results:
          - name: exists
            description: indicates whether the file exists or is missing
        steps:
          - name: check-file
            image: alpine
            script: |
              if test -f $(workspaces.source.path)/$(params.path); then
                printf yes | tee /tekton/results/exists
              else
                printf no | tee /tekton/results/exists
              fi
    - name: echo-file-exists # when expression using task result, evaluates to true
      when:
        - input: "$(tasks.check-file.results.exists)"
          operator: in
          values: ["yes"]
      taskSpec:
        steps:
          - name: echo
            image: ubuntu
            script: 'echo file exists'
    - name: task-should-be-skipped-1
      when:
        - input: "$(tasks.check-file.results.exists)" # when expression using task result, evaluates to false
          operator: in
          values: ["missing"]
      taskSpec:
        steps:
          - name: echo
            image: ubuntu
            script: exit 1
    - name: task-should-be-skipped-2 # when expression using parameter, evaluates to false
      when:
        - input: "$(params.path)"
          operator: notin
          values: ["README.md"]
      taskSpec:
        steps:
          - name: echo
            image: ubuntu
            script: exit 1
    - name: task-should-be-skipped-3 # task with when expression and run after
      runAfter:
        - echo-file-exists
      when:
        - input: "monday"
          operator: in
          values: ["friday"]
      taskSpec:
        steps:
          - name: echo
            image: ubuntu
            script: exit 1
  finally:
    - name: do-something-finally
      taskSpec:
        steps:
          - name: echo
            image: ubuntu
            script: 'echo finally done'

3. update tekton pipeline to version v0.17.1
4. run PipelineRun

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  generateName: guarded-pr-
spec:
  serviceAccountName: 'default'
  pipelineRef:
    name: guarded-pr
  params:
    - name: path
      value: README.md
  workspaces:
    - name: source
      volumeClaimTemplate:
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 16Mi

5. Result

$ kubectl describe pipelinerun guarded-pr-xbjnq
Name:         guarded-pr-xbjnq
Namespace:    tekton-execution
Labels:       tekton.dev/pipeline=guarded-pr
Annotations:  API Version:  tekton.dev/v1beta1
Kind:         PipelineRun
Metadata:
  Creation Timestamp:  2020-10-14T04:21:33Z
  Generate Name:       guarded-pr-
  Generation:          1
  Managed Fields:
    API Version:  tekton.dev/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
        f:labels:
          .:
          f:tekton.dev/pipeline:
      f:status:
        .:
        f:completionTime:
        f:conditions:
        f:pipelineSpec:
          .:
          f:finally:
          f:params:
          f:tasks:
          f:workspaces:
        f:startTime:
    Manager:      controller
    Operation:    Update
    Time:         2020-10-14T04:21:33Z
    API Version:  tekton.dev/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:generateName:
      f:spec:
        .:
        f:params:
        f:pipelineRef:
          .:
          f:name:
        f:serviceAccountName:
        f:workspaces:
    Manager:         kubectl
    Operation:       Update
    Time:            2020-10-14T04:21:33Z
  Resource Version:  489493993
  Self Link:         /apis/tekton.dev/v1beta1/namespaces/tekton-execution/pipelineruns/guarded-pr-xbjnq
  UID:               c17d4135-0700-42fb-b72f-5e4af1064cb5
Spec:
  Params:
    Name:   path
    Value:  README.md
  Pipeline Ref:
    Name:                guarded-pr
  Service Account Name:  default
  Timeout:               1h0m0s
  Workspaces:
    Name:  source
    Volume Claim Template:
      Metadata:
        Creation Timestamp:  <nil>
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:  16Mi
      Status:
Status:
  Completion Time:  2020-10-14T04:21:33Z
  Conditions:
    Last Transition Time:  2020-10-14T04:21:33Z
    Message:               Pipeline tekton-execution/guarded-pr can't be Run; it has an invalid spec: missing field(s): tasks[0].when[0], tasks[2].when[0], tasks[3].when[0], tasks[4].when[0], tasks[5].when[0]
    Reason:                PipelineValidationFailed
    Status:                False
    Type:                  Succeeded
  Pipeline Spec:
    Finally:
      Name:  do-something-finally
      Task Spec:
        Metadata:
        Steps:
          Image:  ubuntu
          Name:   echo
          Resources:
          Script:  echo finally done
    Params:
      Description:  The path of the file to be created
      Name:         path
      Type:         string
    Tasks:
      Name:  create-file
      Task Spec:
        Metadata:
        Steps:
          Image:  ubuntu
          Name:   write-new-stuff
          Resources:
          Script:  touch $(workspaces.source.path)/README.md
        Workspaces:
          Description:  The workspace to create the readme file in
          Name:         source
      When:
        Input:
        Operator:
        Values:    <nil>
      Workspaces:
        Name:       source
        Workspace:  source
      Name:         check-file
      Params:
        Name:   path
        Value:  $(params.path)
      Run After:
        create-file
      Task Spec:
        Metadata:
        Params:
          Name:  path
          Type:  string
        Results:
          Description:  indicates whether the file exists or is missing
          Name:         exists
        Steps:
          Image:  alpine
          Name:   check-file
          Resources:
          Script:  if test -f $(workspaces.source.path)/$(params.path); then
  printf yes | tee /tekton/results/exists
else
  printf no | tee /tekton/results/exists
fi

        Workspaces:
          Description:  The workspace to check for the file
          Name:         source
      Workspaces:
        Name:       source
        Workspace:  source
      Name:         echo-file-exists
      Task Spec:
        Metadata:
        Steps:
          Image:  ubuntu
          Name:   echo
          Resources:
          Script:  echo file exists
      When:
        Input:
        Operator:
        Values:    <nil>
      Name:        task-should-be-skipped-1
      Task Spec:
        Metadata:
        Steps:
          Image:  ubuntu
          Name:   echo
          Resources:
          Script:  exit 1
      When:
        Input:
        Operator:
        Values:    <nil>
      Name:        task-should-be-skipped-2
      Task Spec:
        Metadata:
        Steps:
          Image:  ubuntu
          Name:   echo
          Resources:
          Script:  exit 1
      When:
        Input:
        Operator:
        Values:    <nil>
      Name:        task-should-be-skipped-3
      Run After:
        echo-file-exists
      Task Spec:
        Metadata:
        Steps:
          Image:  ubuntu
          Name:   echo
          Resources:
          Script:  exit 1
      When:
        Input:
        Operator:
        Values:    <nil>
    Workspaces:
      Description:  This workspace is shared among all the pipeline tasks to read/write common resources

      Name:    source
  Start Time:  2020-10-14T04:21:33Z
Events:
  Type     Reason         Age                    From         Message
  ----     ------         ----                   ----         -------
  Normal   Started        4m27s (x2 over 4m27s)  PipelineRun
  Warning  Failed         4m27s (x2 over 4m27s)  PipelineRun  Pipeline tekton-execution/guarded-pr can't be Run; it has an invalid spec: missing field(s): tasks[0].when[0], tasks[2].when[0], tasks[3].when[0], tasks[4].when[0], tasks[5].when[0]
  Warning  InternalError  4m26s (x2 over 4m27s)  PipelineRun  1 error occurred:
           * missing field(s): tasks[0].when[0], tasks[2].when[0], tasks[3].when[0], tasks[4].when[0], tasks[5].when[0]

[vdemeester]

How did the type representing when change ?

[jerop]

How did the type representing when change ?

@vdemeester the only change in the type between v0.16.3 and v0.17.0 was adding json annotations in #3291

[bobcatfish]

Side note that there is some overlap with this issue and tektoncd/triggers#526 - in both cases we are getting some unexpected (and sometimes inconsistent) tolerance of using different casing for fields than what kubernetes is actually storing.

(i.e. since before the change in #3291 k8s was storing the fields starting with uppercase letters, it's kind of confusing that users can still specify them in lower case - if that hadn't been the case we would have noticed this problem right away)

[bobcatfish]

Was brainstorming with @jerop yesterday how we could have avoided this and there didn't seem to be a lot of good options - thinking about it more, I think it would be good to tackle tektoncd/triggers#526 as well and understand more about why we are being tolerant of the capitalization mismatches and try to tackle that - imo that is the root cause of the problem here.

[vdemeester]

The "short" team fix, aside from reverting the change, would be to have Deprecated* fields that use capitalized version of the field and use the default mechanisms to put the right thing at the right place. No matter what we do, there will be, for cluster that were using when in 0.16.x, some object stored with capitalized field.

[ghost]

More brainstorming for options to help catch this in future:

• Lint rule for the API types to ensure json annotations are included on any public fields
• Put the API types in their own package without any additional methods / other types. It may then be easier to apply automation around them - special linting, extra owner reviews required, etc
• Generate OpenAPI docs from the APIs and diff them against existing docs, similarly to how we diff the codegen
  • This could also just be manual: if we see that the API docs were updated we'll realise that the shape of the API has changed?