v2.9.0 workflow-controller OOM Killed #3400

jjtroberts · 2020-07-07T14:46:10Z

Checklist:

I've included the version.
I've included reproduction steps.
I've included the workflow YAML.
I've included the logs.

What happened:
We upgraded from v2.4.3 to v2.9.0 and our heaviest process hung due to the workflow-controller consuming all available memory on the node, resulting in the oom killer reaping the pod +200 times. When I attempted to terminate the workflow, argo responded that it had successfully terminated the workflow, but the status would never change. I had to delete the workflow to remove it altogether. There were no running pods in the client namespace at the time.

What you expected to happen:
The workflow-controller should have remained as stable as it had in v2.4.3.

How to reproduce it (as minimally and precisely as possible):
Run the attached workflow with substitute images using argo-server v2.9.0 on
Kubernetes v1.17.2

Server Version: 18.09.2
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-1127.13.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.51GiB
Name: hqk-prdcp02
ID: AEKM:R3AY:TYBS:LQ74:S55U:3X2Z:CXSE:A4K4:UPLH:TB4K:37I4:C6X3
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

Anything else we need to know?:

Environment:

Argo version:

v2.9.0

Kubernetes version :

v1.17.2

Other debugging information (if applicable):

syslog
messages.txt
workflow-controller logs:
workflow_controller_logs_v2.9.0.txt
workflow template
workflow.txt
node memory stats

workflow-controller stats

You can see where it ran out of memory last night around 10-10:30PM CST, and was terminated, then cron kicked it off again the next morning and it steadily grew until it consumed all available memory.

Message from the maintainers:

If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

The text was updated successfully, but these errors were encountered:

jjtroberts · 2020-07-07T15:06:28Z

Any chance this is related to a fix in v2.9.1?

"fix: Running pods are garaged in PodGC onSuccess"

#3381

alexec · 2020-07-07T15:28:09Z

@jjtroberts you say v2.9.0 in the description, but v2.9.1 in your message. Can you confirm if you have tested with v2.9.0 and v2.9.1 please?

jjtroberts · 2020-07-07T15:57:12Z

I saw the fix in v2.9.1 release only after posting this bug and am currently testing that version. The OOM killer occurred in v2.9.0. I'll update once I know more about v2.9.1 results.

alexec · 2020-07-07T15:59:16Z

Thank you.

jjtroberts · 2020-07-07T17:03:35Z

At the end of the recursive step in v2.9.1, memory still exceeds node capacity:

workflow_controller_logs_v2.9.1.txt

jjtroberts · 2020-07-07T19:47:33Z

I tested with v2.8.2 and the workflow has finally progressed past its point of failure in v2.9.x:

alexec · 2020-07-07T22:13:00Z

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: spring
spec:
  entrypoint: data-warehouse-pipeline

  onExit: exit-handler

  volumes:
    - name: client-volume
      configMap:
        name: client-config
    - name: env-volume
      configMap:
        name: env-config

  templates:

    # TEMPLATES: extractor
    - name: extractor
      inputs:
        parameters:
          - name: integrator-command
          - name: client-code
      container:
        image: docker/whalesay:latest
        imagePullPolicy: IfNotPresent
        command: ["cowsay"]
        # command: ["/bin/bash"]
        # Use "raw" jinja filter so ansible ignores these argo parameters
        # There must be no spaces between the double curly braces and the text or argo will fail to expand.
        args: ["{{inputs.parameters.integrator-command}}", "{{inputs.parameters.client-code}}"]
        # args: ["-c", "while true; do foo; sleep 2; done"]
        resources:
          requests:
            cpu: 200m
            memory: 512Mi
          limits:
            cpu: 800m
            memory: 768Mi
        env:
          - name : APP_ENV
            value: development
          - name : DB_CONNECTION
            value: default
          - name : DB_DRIVER
            value: sqlsrv
          - name : DB_HOST
            value: dwdb01
          - name : DB_USERNAME
            value: gd_det_user
          - name : DB_DATABASE
            value: GD_CRM_EXT_SPRING
          - name : DB_PASSWORD
            valueFrom:
              secretKeyRef:
                key: password
                name: gd-data-extractor-tool-secret
          - name : LSM_DB_HOST
            value: dwdb01
          - name : LSM_DB_DATABASE
            value: GD_LSM_DW
          - name : LSM_DB_USERNAME
            value: gd_det_user
          - name : LSM_DB_PASSWORD
            valueFrom:
              secretKeyRef:
                key: password
                name: gd-data-extractor-tool-secret
        volumeMounts:
          - mountPath: /var/www/html/environments/.env.spring
            name: client-volume
            subPath: .env.spring
          - mountPath: /var/www/html/.env
            name: env-volume
            subPath: .env
    # TEMPLATE: END

    # TEMPLATE: extractor-paginated
    - name: extractor-paginated
      inputs:
        parameters:
          - name: integrator-command
          - name: client-code
          - name: loop-var
      container:
        image: docker/whalesay:latest
        imagePullPolicy: IfNotPresent
        command: ["cowsay"]
        # Use "raw" jinja filter so ansible ignores these argo parameters
        # There must be no spaces between the double curly braces and the text or argo will fail to expand.
        args: ["{{inputs.parameters.integrator-command}}", "{{inputs.parameters.client-code}}", "{{inputs.parameters.loop-var}}"]
        resources:
          requests:
            cpu: 200m
            memory: 512Mi
          limits:
            cpu: 800m
            memory: 768Mi
        env:
          - name : DB_CONNECTION
            value: default
          - name : DB_DRIVER
            value: sqlsrv
          - name : DB_HOST
            value: dwdb01
          - name : DB_USERNAME
            value: gd_det_user
          - name : DB_DATABASE
            value: GD_CRM_EXT_SPRING
          - name : DB_PASSWORD
            valueFrom:
              secretKeyRef:
                key: password
                name: gd-data-extractor-tool-secret
        volumeMounts:
          - mountPath: /var/www/html/environments/.env.spring
            name: client-volume
            subPath: .env.spring
          - mountPath: /var/www/html/.env
            name: env-volume
            subPath: .env
    # TEMPLATE: END

    # TEMPLATE: individual-page-search
    - name: individual-page-search
      script:
        image: docker/whalesay:latest
        imagePullPolicy: IfNotPresent
        command: ["bash"]
        source: |
          echo 100
        # Use "raw" jinja filter so ansible ignores these argo parameters
        # There must be no spaces between the double curly braces and the text or argo will fail to expand.
        resources:
          requests:
            cpu: 200m
            memory: 512Mi
          limits:
            cpu: 800m
            memory: 768Mi
        env:
          - name : DB_CONNECTION
            value: default
          - name : DB_DRIVER
            value: sqlsrv
          - name : DB_HOST
            value: dwdb01
          - name : DB_USERNAME
            value: gd_det_user
          - name : DB_DATABASE
            value: GD_CRM_EXT_SPRING
          - name : DB_PASSWORD
            valueFrom:
              secretKeyRef:
                key: password
                name: gd-data-extractor-tool-secret
        volumeMounts:
          - mountPath: /var/www/html/environments/.env.spring
            name: client-volume
            subPath: .env.spring
          - mountPath: /var/www/html/.env
            name: env-volume
            subPath: .env
    # TEMPLATE: END

    # TEMPLATE: individual-batch-count
    - name: individual-batch-count
      script:
        image: docker/whalesay:latest
        imagePullPolicy: IfNotPresent
        command: ["bash"]
        source: |
          true
        # Use "raw" jinja filter so ansible ignores these argo parameters
        # There must be no spaces between the double curly braces and the text or argo will fail to expand.
        resources:
          requests:
            cpu: 200m
            memory: 512Mi
          limits:
            cpu: 800m
            memory: 768Mi
        env:
          - name : DB_CONNECTION
            value: default
          - name : DB_DRIVER
            value: sqlsrv
          - name : DB_HOST
            value: dwdb01
          - name : DB_USERNAME
            value: gd_det_user
          - name : DB_DATABASE
            value: GD_CRM_EXT_SPRING
          - name : DB_PASSWORD
            valueFrom:
              secretKeyRef:
                key: password
                name: gd-data-extractor-tool-secret
        volumeMounts:
          - mountPath: /var/www/html/environments/.env.spring
            name: client-volume
            subPath: .env.spring
          - mountPath: /var/www/html/.env
            name: env-volume
            subPath: .env
    # TEMPLATE: END

    # TEMPLATE: sqlcmd
    - name: sqlcmd
      inputs:
        parameters:
          - name: sql-command
      container:
        image: docker/whalesay:latest
        imagePullPolicy: IfNotPresent
        command: ["cowsay"]
        # Use "raw" jinja filter so ansible ignores these argo parameters
        # There must be no spaces between the double curly braces and the text or argo will fail to expand.
        args: ["{{inputs.parameters.sql-command}}"]
        resources:
          requests:
            cpu: 200m
            memory: 512Mi
          limits:
            cpu: 200m
            memory: 512Mi
        env:
          - name : ClientId
            value: SPRING
          - name : DB_HOST
            value: dwdb01
          - name : DB_DATABASE
            value: GD_CRM_EXT_SPRING
          - name : DB_USERNAME
            value: gd_det_user
          - name : DB_PASSWORD
            valueFrom:
              secretKeyRef:
                key: password
                name: gd-data-extractor-tool-secret
          - name : DB_INTAKE_USERNAME
            value: gd_intake_user
          - name : DB_INTAKE_PASSWORD
            valueFrom:
              secretKeyRef:
                key: password
                name: gd-data-intake-tool-secret
          - name : DB_FLYWAY_USERNAME
            value: svc_datawh_flyway_dev
          - name : DB_FLYWAY_PASSWORD
            valueFrom:
              secretKeyRef:
                key: password
                name: mssqlserver-mssql-linux-secret
    # TEMPLATE: END

    # TEMPLATE: whalesay
    - name: whalesay
      inputs:
        parameters:
          - name: message
      container:
        image: docker/whalesay:latest
        command: [cowsay]
        args: ["{{inputs.parameters.message}}"]
    # TEMPLATE: END

    # TEMPLATE: exit-handler
    - name: exit-handler
      steps:
        - - name: notify
            template: echo
            when: "{{workflow.status}} != Succeeded"
    # TEMPLATE: END

    # TEMPLATE: echo
    - name: echo
      container:
        image: docker/whalesay:latest
        command: [sh, -c]
        args: ["echo {{workflow.name}} {{workflow.status}}"]
    # TEMPLATE: END

    # TEMPLATE: calculate-loop-end
    - name: calculate-loop-end
      inputs:
        parameters:
          - name: total
          - name: limit
      script:
        image: python:latest
        command: [python]
        args: ["-c", "print(({{inputs.parameters.total}} - {{inputs.parameters.limit}}) + 1)"]
    # TEMPLATE: END

    # TEMPLATE: subtracter
    - name: subtracter
      inputs:
        parameters:
          - name: total
          - name: limit
      script:
        image: python:latest
        command: [python]
        args: ["-c", "print(int((abs(({{inputs.parameters.total}}-{{inputs.parameters.limit}}))+({{inputs.parameters.total}}-{{inputs.parameters.limit}}))/2))"]
    # TEMPLATE: END

    # TEMPLATE: pull-individual-page-repeatable
    - name: pull-individual-page-repeatable
      inputs:
        parameters:
          - name: total
          - name: limit
          - name: integrator-command
          - name: client-code
      dag:
        tasks:

          - name: get-loop-end-var
            template: calculate-loop-end
            arguments:
              parameters:
                - name: total
                  value: "{{inputs.parameters.total}}"
                - name: limit
                  value: "{{inputs.parameters.limit}}"

          - name: PullIndividualPage-AZ
            dependencies: [get-loop-end-var]
            template: extractor-paginated
            arguments:
              parameters:
                - name: integrator-command
                  value: "{{inputs.parameters.integrator-command}}"
                - name: client-code
                  value: "{{inputs.parameters.client-code}}"
                - name: loop-var
                  value: "{{item}}"
            withSequence:
              start: "{{inputs.parameters.total}}"
              end  : "{{tasks.get-loop-end-var.outputs.result}}"
            when: "{{tasks.get-loop-end-var.outputs.result}} > 0"

          - name: PullIndividualPage-BZ
            dependencies: [get-loop-end-var]
            template: extractor-paginated
            arguments:
              parameters:
                - name: integrator-command
                  value: "{{inputs.parameters.integrator-command}}"
                - name: client-code
                  value: "{{inputs.parameters.client-code}}"
                - name: loop-var
                  value: "{{item}}"
            withSequence:
              start: "{{inputs.parameters.total}}"
              end  : "1"
            when: "{{tasks.get-loop-end-var.outputs.result}} <= 0"

          - name: get-total-minus-limit
            dependencies: [PullIndividualPage-AZ, PullIndividualPage-BZ]
            template: subtracter
            arguments:
              parameters:
                - name: total
                  value: "{{inputs.parameters.total}}"
                - name: limit
                  value: "{{inputs.parameters.limit}}"

          - name: do-the-wave
            dependencies: [get-total-minus-limit]
            template: pull-individual-page-repeatable
            arguments:
              parameters:
                - name: total
                  value: "{{tasks.get-total-minus-limit.outputs.result}}"
                - name: limit
                  value: "{{inputs.parameters.limit}}"
                - name: integrator-command
                  value: "{{inputs.parameters.integrator-command}}"
                - name: client-code
                  value: "{{inputs.parameters.client-code}}"
            when: "{{tasks.get-total-minus-limit.outputs.result}} > 0"
    # TEMPLATE: END

    # TEMPLATE: data-warehouse-pipeline
    - name: data-warehouse-pipeline
      dag:
        tasks:

          - name: Migrate
            template: extractor
            arguments:
              parameters:
                - name: integrator-command
                  value: "run-migration"
                - name: client-code
                  value: "spring"

          - name: IndividualPageSearch
            dependencies: [Migrate]
            template: individual-page-search

          - name: PullIndividualPageByWave
            dependencies: [IndividualPageSearch]
            template: pull-individual-page-repeatable
            arguments:
              parameters:
                - name: total
                  value: "{{tasks.IndividualPageSearch.outputs.result}}"
                - name: limit
                  value: "4"
                - name: integrator-command
                  value: "pull:individual-page"
                - name: client-code
                  value: "spring"

          - name: IndividualBatchCount
            dependencies: [PullIndividualPageByWave]
            template: individual-batch-count

          - name: PullActivitiesCasesIndividualBatch
            dependencies: [IndividualBatchCount]
            template: extractor-paginated
            arguments:
              parameters:
                - name: integrator-command
                  value: "pull:activites-cases-individual-batch"
                - name: client-code
                  value: "spring"
                - name: loop-var
                  value: "{{item}}"
            withSequence:
              #count: "{{tasks.IndividualBatchCount.outputs.result}}"
              start: "1"
              end: "{{tasks.IndividualBatchCount.outputs.result}}"

          - name: ConsolidateBatches
            dependencies: [PullActivitiesCasesIndividualBatch]
            template: whalesay
            arguments:
              parameters:
                - name: message
                  value: "PullActivitiesCasesIndividualBatch Completed"

          - name: Address
            dependencies: [ConsolidateBatches]
            template: extractor
            arguments:
              parameters:
                - name: integrator-command
                  value: "pull:address"
                - name: client-code
                  value: "spring"

          - name: HousingContracts
            dependencies: [ConsolidateBatches]
            template: extractor
            arguments:
              parameters:
                - name: integrator-command
                  value: "pull:housing_contracts"
                - name: client-code
                  value: "spring"

          - name: Transactions
            dependencies: [ConsolidateBatches]
            template: extractor
            arguments:
              parameters:
                - name: integrator-command
                  value: "pull:transactions"
                - name: client-code
                  value: "spring"

          - name: Campaigns
            dependencies: [ConsolidateBatches]
            template: extractor
            arguments:
              parameters:
                - name: integrator-command
                  value: "pull:campaigns-and-campaign-individuals"
                - name: client-code
                  value: "spring"

          - name: Communities
            dependencies: [ConsolidateBatches]
            template: extractor
            arguments:
              parameters:
                - name: integrator-command
                  value: "pull:communities"
                - name: client-code
                  value: "spring"

          - name: Types
            dependencies: [ConsolidateBatches]
            template: extractor
            arguments:
              parameters:
                - name: integrator-command
                  value: "pull:types"
                - name: client-code
                  value: "spring"

          - name: Units
            dependencies: [ConsolidateBatches]
            template: extractor
            arguments:
              parameters:
                - name: integrator-command
                  value: "pull:units"
                - name: client-code
                  value: "spring"

          - name: Users
            dependencies: [ConsolidateBatches]
            template: extractor
            arguments:
              parameters:
                - name: integrator-command
                  value: "pull:users"
                - name: client-code
                  value: "spring"

          - name: ActOn
            dependencies: [ConsolidateBatches]
            template: pull-from-act-on

          - name: RunIntake
            dependencies: [Address, HousingContracts, Transactions, Campaigns, Communities, Types, Units, Users, ActOn]
            template: sqlcmd
            arguments:
              parameters:
                - name: sql-command
                  value: /opt/mssql-tools/bin/sqlcmd -b -S $DB_HOST -U $DB_INTAKE_USERNAME -P $DB_INTAKE_PASSWORD -i /opt/scripts/1_run_usp_RunIntake.sql

          - name: RemoveDuplicateIndividuals
            dependencies: [RunIntake]
            template: sqlcmd
            arguments:
              parameters:
                - name: sql-command
                  value: /opt/mssql-tools/bin/sqlcmd -b -S $DB_HOST -U $DB_INTAKE_USERNAME -P $DB_INTAKE_PASSWORD -i /opt/scripts/2_run_usp_RemoveDuplicateIndividuals.sql

          - name: PopulateLookupsInStage
            dependencies: [RemoveDuplicateIndividuals]
            template: sqlcmd
            arguments:
              parameters:
                - name: sql-command
                  value: /opt/mssql-tools/bin/sqlcmd -b -S $DB_HOST -U $DB_INTAKE_USERNAME -P $DB_INTAKE_PASSWORD -i /opt/scripts/3_run_usp_PopulateLookupsInStage.sql

          - name: SalesCounselorStageToDW
            dependencies: [PopulateLookupsInStage]
            template: sqlcmd
            arguments:
              parameters:
                - name: sql-command
                  value: /opt/mssql-tools/bin/sqlcmd -b -S $DB_HOST -U $DB_INTAKE_USERNAME -P $DB_INTAKE_PASSWORD -i /opt/scripts/4_run_usp_SalesCounselorStageToDW.sql

          - name: AccountStageToDW
            dependencies: [SalesCounselorStageToDW]
            template: sqlcmd
            arguments:
              parameters:
                - name: sql-command
                  value: /opt/mssql-tools/bin/sqlcmd -b -S $DB_HOST -U $DB_INTAKE_USERNAME -P $DB_INTAKE_PASSWORD -i /opt/scripts/5_run_usp_AccountStageToDW.sql

          - name: OpportunityStageToDW
            dependencies: [AccountStageToDW]
            template: sqlcmd
            arguments:
              parameters:
                - name: sql-command
                  value: /opt/mssql-tools/bin/sqlcmd -b -S $DB_HOST -U $DB_INTAKE_USERNAME -P $DB_INTAKE_PASSWORD -i /opt/scripts/6_run_usp_OpportunityStageToDW.sql

          - name: IndividualStageToDW
            dependencies: [OpportunityStageToDW]
            template: sqlcmd
            arguments:
              parameters:
                - name: sql-command
                  value: /opt/mssql-tools/bin/sqlcmd -b -S $DB_HOST -U $DB_INTAKE_USERNAME -P $DB_INTAKE_PASSWORD -i /opt/scripts/7_run_usp_IndividualStageToDW.sql

          - name: ActivityStageToDW
            dependencies: [IndividualStageToDW]
            template: sqlcmd
            arguments:
              parameters:
                - name: sql-command
                  value: /opt/mssql-tools/bin/sqlcmd -b -S $DB_HOST -U $DB_INTAKE_USERNAME -P $DB_INTAKE_PASSWORD -i /opt/scripts/8_run_usp_ActivityStageToDW.sql

          - name: ProcessCRMDWScores
            dependencies: [ActivityStageToDW]
            template: sqlcmd
            arguments:
              parameters:
                - name: sql-command
                  value: /opt/mssql-tools/bin/sqlcmd -b -S $DB_HOST -U $DB_INTAKE_USERNAME -P $DB_INTAKE_PASSWORD -i /opt/scripts/9_run_usp_Process_CRMDW_Scores.sql

          - name: Push
            dependencies: [ProcessCRMDWScores]
            template: extractor
            arguments:
              parameters:
                - name: integrator-command
                  value: "push:scores"
                - name: client-code
                  value: "spring"
    # TEMPLATE: END

    # TEMPLATE: pull-from-act-on
    - name: pull-from-act-on
      script:
        image: docker/whalesay:latest
        imagePullPolicy: IfNotPresent
        command: ["bash"]
        source: |
          echo "pull_from_act_on var was false; doing nothing"
        # Use "raw" jinja filter so ansible ignores these argo parameters
        # There must be no spaces between the double curly braces and the text or argo will fail to expand.
        resources:
          requests:
            cpu: 200m
            memory: 512Mi
          limits:
            cpu: 800m
            memory: 768Mi
        env:
          - name : DB_CONNECTION
            value: default
          - name : DB_DRIVER
            value: sqlsrv
          - name : DB_HOST
            value: dwdb01
          - name : DB_USERNAME
            value: gd_det_user
          - name : DB_DATABASE
            value: GD_CRM_EXT_SPRING
          - name : DB_PASSWORD
            valueFrom:
              secretKeyRef:
                key: password
                name: gd-data-extractor-tool-secret
        volumeMounts:
          - mountPath: /var/www/html/environments/.env.spring
            name: client-volume
            subPath: .env.spring
          - mountPath: /var/www/html/.env
            name: env-volume
            subPath: .env
    # TEMPLATE: END

@simster7

alexec · 2020-07-07T22:13:11Z

     640MB 94.89% 94.89%   663.01MB 98.30%  github.com/argoproj/argo/workflow/controller.(*dagContext).assessDAGPhase
   21.01MB  3.11% 98.00%    21.01MB  3.11%  github.com/argoproj/argo/workflow/controller.getChildNodeIndex

alexec · 2020-07-07T22:15:05Z

      flat  flat%   sum%        cum   cum%
    1280MB 98.58% 98.58%     1287MB 99.12%  github.com/argoproj/argo/workflow/controller.(*dagContext).assessDAGPhase

alexec · 2020-07-07T22:15:11Z

https://www.freecodecamp.org/news/how-i-investigated-memory-leaks-in-go-using-pprof-on-a-large-codebase-4bec4325e192/

alexec · 2020-07-07T22:15:55Z

alexec · 2020-07-07T22:39:09Z

spring.yaml.txt

alexec · 2020-07-08T22:45:05Z

Fix in v2.9.2

jjtroberts added the type/bug label Jul 7, 2020

alexec self-assigned this Jul 7, 2020

alexec added this to the v2.9 milestone Jul 7, 2020

alexec mentioned this issue Jul 7, 2020

v2.8 vs v2.9 #3406

Closed

This was linked to pull requests Jul 8, 2020

fix: Use a unique queue to visit nodes #3418

Merged

fix: Add struct-wide RWMutex to metrics #3421

Merged

alexec removed a link to a pull request Jul 8, 2020

fix: Use a unique queue to visit nodes #3418

Merged

alexec closed this as completed Jul 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v2.9.0 workflow-controller OOM Killed #3400

v2.9.0 workflow-controller OOM Killed #3400

jjtroberts commented Jul 7, 2020 •

edited

Loading

jjtroberts commented Jul 7, 2020 •

edited

Loading

alexec commented Jul 7, 2020

jjtroberts commented Jul 7, 2020

alexec commented Jul 7, 2020

jjtroberts commented Jul 7, 2020 •

edited

Loading

jjtroberts commented Jul 7, 2020

alexec commented Jul 7, 2020

alexec commented Jul 7, 2020

alexec commented Jul 7, 2020

alexec commented Jul 7, 2020

alexec commented Jul 7, 2020

alexec commented Jul 7, 2020

alexec commented Jul 8, 2020

v2.9.0 workflow-controller OOM Killed #3400

v2.9.0 workflow-controller OOM Killed #3400

Comments

jjtroberts commented Jul 7, 2020 • edited Loading

jjtroberts commented Jul 7, 2020 • edited Loading

alexec commented Jul 7, 2020

jjtroberts commented Jul 7, 2020

alexec commented Jul 7, 2020

jjtroberts commented Jul 7, 2020 • edited Loading

jjtroberts commented Jul 7, 2020

alexec commented Jul 7, 2020

alexec commented Jul 7, 2020

alexec commented Jul 7, 2020

alexec commented Jul 7, 2020

alexec commented Jul 7, 2020

alexec commented Jul 7, 2020

alexec commented Jul 8, 2020

jjtroberts commented Jul 7, 2020 •

edited

Loading

jjtroberts commented Jul 7, 2020 •

edited

Loading

jjtroberts commented Jul 7, 2020 •

edited

Loading