support optional Input artifacts when artifacts not exist

[haibingzhao]

Summary

support optional Input artifacts when artifacts not exist.

Motivation

1. example_a output condition_b1:true and condition_b2:false
2. condition_1 satisfied and execute exapmle_b, while condition_2 not satisfied
3. example_c get condition_b1 and condition_b2 outputs with optional field:

 inputs:
      artifacts:
      - {name: example-b-result_b, optional: true,  path: /tmp/inputs/result_b1/data}
      - {name: example-b-2-result_b,  optional: true, path: /tmp/inputs/result_b2/data}

because condition_2 not satisfied, so get error:
Unable to resolve: {{tasks.condition-2.outputs.artifacts.example-b-2-result_b}}

detail workflow yaml:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: test-
spec:
  entrypoint: test
  templates:
  - name: condition-1
    outputs:
      artifacts:
      - {name: example-b-result_b, from: '{{tasks.example-b.outputs.artifacts.example-b-result_b}}'}
    dag:
      tasks:
      - {name: example-b, template: example-b}
  - name: condition-2
    outputs:
      artifacts:
      - {name: example-b-2-result_b, from: '{{tasks.example-b-2.outputs.artifacts.example-b-2-result_b}}'}
    dag:
      tasks:
      - {name: example-b-2, template: example-b-2}
  - name: example-a
    container:
      args: [components.train.test.runa, run, --condition_b1, /tmp/outputs/condition_b1/data,
        --condition_b2, /tmp/outputs/condition_b2/data]
      command: [python, -u, -m, components.launcher]
      image: xxx:test
      workingDir: /pipelines
      volumeMounts:
      - {mountPath: /pipelines, name: pipeline}
    outputs:
      parameters:
      - name: example-a-condition_b1
        valueFrom: {path: /tmp/outputs/condition_b1/data}
      - name: example-a-condition_b2
        valueFrom: {path: /tmp/outputs/condition_b2/data}
      artifacts:
      - {name: example-a-condition_b1, path: /tmp/outputs/condition_b1/data}
      - {name: example-a-condition_b2, path: /tmp/outputs/condition_b2/data}
    volumes:
    - hostPath: {path: /pipeline, type: Directory}
      name: pipeline
  - name: example-b
    container:
      args: [components.train.test.runb, run, --type_b, bbb1, --result_b, /tmp/outputs/result_b/data]
      command: [python, -u, -m, components.launcher]
      image: xxx:test
      workingDir: /pipelines
      volumeMounts:
      - {mountPath: /pipelines, name: pipeline}
    outputs:
      artifacts:
      - {name: example-b-result_b, path: /tmp/outputs/result_b/data}
    volumes:
    - hostPath: {path: /pipeline, type: Directory}
      name: pipeline
  - name: example-b-2
    container:
      args: [components.train.test.runb, run, --type_b, bbb2, --result_b, /tmp/outputs/result_b/data]
      command: [python, -u, -m, components.launcher]
      image: xxx:test
      workingDir: /pipelines
      volumeMounts:
      - {mountPath: /pipelines, name: pipeline}
    outputs:
      artifacts:
      - {name: example-b-2-result_b, path: /tmp/outputs/result_b/data}
    volumes:
    - hostPath: {path: /pipeline, type: Directory}
      name: pipeline
  - name: example-c
    container:
      args: [components.train.test.runc, run, --result_b1, /tmp/inputs/result_b1/data,
        --result_b2, /tmp/inputs/result_b2/data]
      command: [python, -u, -m, components.launcher]
      image: xxx:test
      workingDir: /pipelines
      volumeMounts:
      - {mountPath: /pipelines, name: pipeline}
    inputs:
      artifacts:
      - {name: example-b-result_b, optional: true,  path: /tmp/inputs/result_b1/data}
      - {name: example-b-2-result_b,  optional: true, path: /tmp/inputs/result_b2/data}
    volumes:
    - hostPath: {path: /pipeline, type: Directory}
      name: pipeline
  - name: test
    dag:
      tasks:
      - name: condition-1
        template: condition-1
        when: '"{{tasks.example-a.outputs.parameters.example-a-condition_b1}}" ==
          "true"'
        dependencies: [example-a]
      - name: condition-2
        template: condition-2
        when: '"{{tasks.example-a.outputs.parameters.example-a-condition_b2}}" ==
          "true"'
        dependencies: [example-a]
      - {name: example-a, template: example-a}
      - name: example-c
        template: example-c
        dependencies: [condition-1, condition-2]
        arguments:
          artifacts:
          - {name: example-b-2-result_b, from: '{{tasks.condition-2.outputs.artifacts.example-b-2-result_b}}'}
          - {name: example-b-result_b, from: '{{tasks.condition-1.outputs.artifacts.example-b-result_b}}'}
  arguments:
    parameters: []

Proposal

How do you think this should be implemented?

when artifact not exist and specified with optional: true : return nil instead report error.

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[simster7]

What version are you running? This might be fixed already

[alexec]

+1 @simster7 #3019 fixed in v2.9

[haibingzhao]

+1 @simster7 #3019 fixed in v2.9

running with latest version not work in my case. you can submit below wf to reproduce or check argo-issue-example repository :

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: test-
spec:
  entrypoint: test
  templates:
  - name: condition-1
    outputs:
      artifacts:
      - {name: example-b-result_b, from: '{{tasks.example-b.outputs.artifacts.example-b-result_b}}'}
    dag:
      tasks:
      - {name: example-b, template: example-b}
  - name: condition-2
    outputs:
      artifacts:
      - {name: example-b-2-result_b, from: '{{tasks.example-b-2.outputs.artifacts.example-b-2-result_b}}'}
    dag:
      tasks:
      - {name: example-b-2, template: example-b-2}
  - name: example-a
    container:
      args: [runa, --condition_b1, /tmp/outputs/condition_b1/data,
        --condition_b2, /tmp/outputs/condition_b2/data]
      command: [python, -u, -m]
      image: ascpforecasting/test:0.0.1
      workingDir: /pipelines
    outputs:
      parameters:
      - name: example-a-condition_b1
        valueFrom: {path: /tmp/outputs/condition_b1/data}
      - name: example-a-condition_b2
        valueFrom: {path: /tmp/outputs/condition_b2/data}
      artifacts:
      - {name: example-a-condition_b1, path: /tmp/outputs/condition_b1/data}
      - {name: example-a-condition_b2, path: /tmp/outputs/condition_b2/data}
  - name: example-b
    container:
      args: [runb, --type_b, bbb1, --result_b, /tmp/outputs/result_b/data]
      command: [python, -u, -m]
      image: ascpforecasting/test:0.0.1
      workingDir: /pipelines
    outputs:
      artifacts:
      - {name: example-b-result_b, path: /tmp/outputs/result_b/data}
  - name: example-b-2
    container:
      args: [runb, --type_b, bbb2, --result_b, /tmp/outputs/result_b/data]
      command: [python, -u, -m]
      image: ascpforecasting/test:0.0.1
      workingDir: /pipelines
    outputs:
      artifacts:
      - {name: example-b-2-result_b, path: /tmp/outputs/result_b/data}
  - name: example-c
    container:
      args: [runc, --result_b1, /tmp/inputs/result_b1/data,
        --result_b2, /tmp/inputs/result_b2/data]
      command: [python, -u, -m]
      image: ascpforecasting/test:0.0.1
      workingDir: /pipelines
    inputs:
      artifacts:
      - {name: example-b-result_b, optional: true,  path: /tmp/inputs/result_b1/data}
      - {name: example-b-2-result_b,  optional: true, path: /tmp/inputs/result_b2/data}
  - name: test
    dag:
      tasks:
      - name: condition-1
        template: condition-1
        when: '"{{tasks.example-a.outputs.parameters.example-a-condition_b1}}" ==
          "true"'
        dependencies: [example-a]
      - name: condition-2
        template: condition-2
        when: '"{{tasks.example-a.outputs.parameters.example-a-condition_b2}}" ==
          "true"'
        dependencies: [example-a]
      - {name: example-a, template: example-a}
      - name: example-c
        template: example-c
        dependencies: [condition-1, condition-2]
        arguments:
          artifacts:
          - {name: example-b-2-result_b, optional: true, from: '{{tasks.condition-2.outputs.artifacts.example-b-2-result_b}}'}
          - {name: example-b-result_b, optional: true, from: '{{tasks.condition-1.outputs.artifacts.example-b-result_b}}'}
  arguments:
    parameters: []

after search source code, I found might be a little code change at: workflow/controller/dag.go#L539, and it works now.

[simster7]

I'm able to reproduce. Found a fix

[simster7]

@haibingzhao, I just noticed you found the same fix. Would you like to open a PR?

[haibingzhao]

@haibingzhao, I just noticed you found the same fix. Would you like to open a PR?

opened PR can be found here: #3512