Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Template parallelism not limiting nested workflows #1035

Closed
WeiTang114 opened this issue Oct 11, 2018 · 1 comment
Closed

Template parallelism not limiting nested workflows #1035

WeiTang114 opened this issue Oct 11, 2018 · 1 comment

Comments

@WeiTang114
Copy link
Contributor

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT (or is it intended behavior?)

What happened:
Template parallelism is not limiting nested workflows when specified on the outer workflow.
I have a workflow template A. It has steps with withParam to run N times of workflow template B. B has steps with withParam to run several times of a container template one-step.
I set parallelism: 1 on A, and I expect to run only one B at a time.

What you expected to happen:
Only one B is run at a time.

How to reproduce it (as minimally and precisely as possible):
This is modified from parallelism-nested.yaml example.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: seq-test-
spec:
  arguments:
    parameters:
    - name: seq-list
      value: |
        ["a","b","c","d"]
  entrypoint: A
  templates:
  - name: A
    parallelism: 1
    inputs:
      parameters:
      - name: seq-list
    steps:
    - - name: seq-step
        template: B
        arguments:
          parameters:
          - name: seq-id
            value: "{{item}}"
        withParam: "{{inputs.parameters.seq-list}}"

  - name: B
    inputs:
      parameters:
      - name: seq-id
    steps:
    - - name: jobs
        template: one-job
        arguments:
          parameters:
          - name: seq-id
            value: "{{inputs.parameters.seq-id}}"
        withParam: "[1, 2]"


  - name: one-job
    inputs:
      parameters:
      - name: seq-id
    container:
      image: alpine
      command: ['/bin/sh', '-c']
      args: ["echo {{inputs.parameters.seq-id}}; sleep 10"]

Anything else we need to know?:

Environment:

  • Argo version:
argo: v2.2.0
  BuildDate: 2018-08-30T08:51:40Z
  GitCommit: af636ddd8455660f307d835814d3112b90815dfd
  GitTreeState: clean
  GitTag: v2.2.0
  GoVersion: go1.10.3
  Compiler: gc
  Platform: darwin/amd64
  • Kubernetes version :
clientVersion:
  buildDate: 2018-09-10T11:44:26Z
  compiler: gc
  gitCommit: a4529464e4629c21224b3d52edfe0ea91b072862
  gitTreeState: clean
  gitVersion: v1.11.3
  goVersion: go1.11
  major: "1"
  minor: "11"
  platform: darwin/amd64
serverVersion:
  buildDate: 2018-09-13T22:19:55Z
  compiler: gc
  gitCommit: 8d9503f982872112eb283f78cefc6944af640427
  gitTreeState: clean
  gitVersion: v1.10.7-gke.2
  goVersion: go1.9.3b4
  major: "1"
  minor: 10+
  platform: linux/amd64

Other debugging information (if applicable):

  • workflow result:
 $ argo get seq-test-bbm5v                                                                                                                                                                       [11:53:36]
Name:                seq-test-bbm5v
Namespace:           default
ServiceAccount:      default
Status:              Succeeded
Created:             Thu Oct 11 11:50:31 +0800 (3 minutes ago)
Started:             Thu Oct 11 11:50:31 +0800 (3 minutes ago)
Finished:            Thu Oct 11 11:51:00 +0800 (2 minutes ago)
Duration:            29 seconds
Parameters:
  seq-list:          ["a","b","c","d"]


STEP                  PODNAME                    DURATION  MESSAGE
 ✔ seq-test-bbm5v
 └-·-✔ seq-step(0:a)
   | └-·-✔ jobs(0:1)  seq-test-bbm5v-1283787062  12s
   |   └-✔ jobs(1:2)  seq-test-bbm5v-556325212   12s
   ├-✔ seq-step(1:b)
   | └-·-✔ jobs(0:1)  seq-test-bbm5v-2674682628  12s
   |   └-✔ jobs(1:2)  seq-test-bbm5v-2770338894  12s
   ├-✔ seq-step(2:c)
   | └-·-✔ jobs(0:1)  seq-test-bbm5v-554334974   13s
   |   └-✔ jobs(1:2)  seq-test-bbm5v-1762128036  13s
   └-✔ seq-step(3:d)
     └-·-✔ jobs(0:1)  seq-test-bbm5v-64729936    13s
       └-✔ jobs(1:2)  seq-test-bbm5v-3655250666  13s

WeiTang114 added a commit to WeiTang114/argo that referenced this issue Oct 16, 2018
Related issue: argoproj#1035

What is solved:
 - Parallelism for nested steps (StepGroups)

What is not solved:
 - Parallelism for nested DAG

This commit make nested StepGroup workflow parallelism on the outer
workflow limiting inner workflow execution. This is done by making
checkParallelism called by the inner workflow checks if the number of
its running siblings (the nodes with the same parent node) is >=
the parent node's parallelism.
@WeiTang114
Copy link
Contributor Author

Fixed by #1046. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant