Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Workflow fails when artifact file exists, but is empty #1642

Closed
Ark-kun opened this issue Oct 4, 2019 · 4 comments · Fixed by #1653
Closed

[Bug] Workflow fails when artifact file exists, but is empty #1642

Ark-kun opened this issue Oct 4, 2019 · 4 comments · Fixed by #1653

Comments

@Ark-kun
Copy link
Member

Ark-kun commented Oct 4, 2019

Is this a BUG REPORT or FEATURE REQUEST?:

BUG REPORT

What happened:
Workflow fails when output artifact file exists, but is empty.

It's a blocking issue for us since we have tasks where some outputs are often empty or some mutually exclusive outputs.

What you expected to happen:

Empty output files should be a valid scenario.

How to reproduce it (as minimally and precisely as possible):

argo submit --namespace kubeflow --serviceaccount pipeline-runner /dev/stdin <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: produce-empty-
spec:
  entrypoint: produce-empty
  templates:
  - name: produce-empty
    outputs:
      artifacts:
      - name: out
        path: /tmp/out.txt
    container:
      image: busybox
      command: [touch, /tmp/out.txt]
EOF

Anything else we need to know?:

Environment:

  • Argo version:
2.3.0

Other debugging information (if applicable):

  • workflow result:
$ argo get --namespace kubeflow produce-empty-gpw5k
Name:                produce-empty-gpw5k
Namespace:           kubeflow
ServiceAccount:      pipeline-runner
Status:              Error
Message:             failed to save outputs: path /tmp/out.txt does not exist (or /tmp/out.txt is empty) in archive /argo/outputs/artifacts/out.tgz
Created:             Thu Oct 03 19:08:37 -0700 (8 seconds ago)
Started:             Thu Oct 03 19:08:37 -0700 (8 seconds ago)
Finished:            Thu Oct 03 19:08:42 -0700 (3 seconds ago)
Duration:            5 seconds

STEP                    PODNAME              DURATION  MESSAGE
 ⚠ produce-empty-gpw5k  produce-empty-gpw5k  4s        failed to save outputs: path /tmp/out.txt does not exist (or /tmp/out.txt is empty) in archive /argo/outputs/artifacts/out.tgz
  • executor logs:
$ kubectl logs produce-empty-gpw5k -c wait --namespace kubeflow
time="2019-10-04T02:08:39Z" level=info msg="Creating a docker executor"
time="2019-10-04T02:08:39Z" level=info msg="Executor (version: v2.3.0, build_date: 2019-05-20T22:10:54Z) initialized (pod: kubeflow/produce-empty-gpw5k) with template:\n{\"name\":\"produce-empty\",\"inputs\":{},\"outputs\":{\"artifacts\":[{\"name\":\"out\",\"path\":\"/tmp/out.txt\"}]},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"busybox\",\"command\":[\"touch\",\"/tmp/out.txt\"],\"resources\":{}},\"archiveLocation\":{\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"key\":\"artifacts/produce-empty-gpw5k/produce-empty-gpw5k\"}}}"
time="2019-10-04T02:08:39Z" level=info msg="Waiting on main container"
time="2019-10-04T02:08:40Z" level=info msg="main container started with container ID: d5548aee01fe0f69695ba6990d3c9f56666d5c07b7be2d399398e8c9deabc246"
time="2019-10-04T02:08:40Z" level=info msg="Starting annotations monitor"
time="2019-10-04T02:08:40Z" level=info msg="Starting deadline monitor"
time="2019-10-04T02:08:40Z" level=info msg="docker wait d5548aee01fe0f69695ba6990d3c9f56666d5c07b7be2d399398e8c9deabc246"
time="2019-10-04T02:08:41Z" level=info msg="Main container completed"
time="2019-10-04T02:08:41Z" level=info msg="No sidecars"
time="2019-10-04T02:08:41Z" level=info msg="No output parameters"
time="2019-10-04T02:08:41Z" level=info msg="Saving output artifacts"
time="2019-10-04T02:08:41Z" level=info msg="Annotations monitor stopped"
time="2019-10-04T02:08:41Z" level=info msg="Staging artifact: out"
time="2019-10-04T02:08:41Z" level=info msg="Copying /tmp/out.txt from container base image layer to /argo/outputs/artifacts/out.tgz"
time="2019-10-04T02:08:41Z" level=info msg="Archiving d5548aee01fe0f69695ba6990d3c9f56666d5c07b7be2d399398e8c9deabc246:/tmp/out.txt to /argo/outputs/artifacts/out.tgz"
time="2019-10-04T02:08:41Z" level=info msg="sh -c docker cp -a d5548aee01fe0f69695ba6990d3c9f56666d5c07b7be2d399398e8c9deabc246:/tmp/out.txt - | gzip > /argo/outputs/artifacts/out.tgz"
time="2019-10-04T02:08:41Z" level=warning msg="path /tmp/out.txt does not exist (or /tmp/out.txt is empty) in archive /argo/outputs/artifacts/out.tgz"
time="2019-10-04T02:08:41Z" level=error msg="executor error: path /tmp/out.txt does not exist (or /tmp/out.txt is empty) in archive /argo/outputs/artifacts/out.tgz\ngh.neting.cc/argoproj/argo/errors.New\n\t/go/src/github.com/argoproj/argo/errors/errors.go:49\ngh.neting.cc/argoproj/argo/errors.Errorf\n\t/go/src/github.com/argoproj/argo/errors/errors.go:55\ngh.neting.cc/argoproj/argo/workflow/executor/docker.(*DockerExecutor).CopyFile\n\t/go/src/github.com/argoproj/argo/workflow/executor/docker/docker.go:66\ngh.neting.cc/argoproj/argo/workflow/executor.(*WorkflowExecutor).stageArchiveFile\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:344\ngh.neting.cc/argoproj/argo/workflow/executor.(*WorkflowExecutor).saveArtifact\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:245\ngh.neting.cc/argoproj/argo/workflow/executor.(*WorkflowExecutor).SaveArtifacts\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:231\ngh.neting.cc/argoproj/argo/cmd/argoexec/commands.waitContainer\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:54\ngh.neting.cc/argoproj/argo/cmd/argoexec/commands.NewWaitCommand.func1\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:16\ngh.neting.cc/spf13/cobra.(*Command).execute\n\t/go/src/github.com/spf13/cobra/command.go:766\ngh.neting.cc/spf13/cobra.(*Command).ExecuteC\n\t/go/src/github.com/spf13/cobra/command.go:852\ngh.neting.cc/spf13/cobra.(*Command).Execute\n\t/go/src/github.com/spf13/cobra/command.go:800\nmain.main\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:17\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:201\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1333"
time="2019-10-04T02:08:41Z" level=info msg="Alloc=4438 TotalAlloc=11090 Sys=70590 NumGC=4 Goroutines=8"
time="2019-10-04T02:08:41Z" level=fatal msg="path /tmp/out.txt does not exist (or /tmp/out.txt is empty) in archive /argo/outputs/artifacts/out.tgz\ngh.neting.cc/argoproj/argo/errors.New\n\t/go/src/github.com/argoproj/argo/errors/errors.go:49\ngh.neting.cc/argoproj/argo/errors.Errorf\n\t/go/src/github.com/argoproj/argo/errors/errors.go:55\ngh.neting.cc/argoproj/argo/workflow/executor/docker.(*DockerExecutor).CopyFile\n\t/go/src/github.com/argoproj/argo/workflow/executor/docker/docker.go:66\ngh.neting.cc/argoproj/argo/workflow/executor.(*WorkflowExecutor).stageArchiveFile\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:344\ngh.neting.cc/argoproj/argo/workflow/executor.(*WorkflowExecutor).saveArtifact\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:245\ngh.neting.cc/argoproj/argo/workflow/executor.(*WorkflowExecutor).SaveArtifacts\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:231\ngh.neting.cc/argoproj/argo/cmd/argoexec/commands.waitContainer\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:54\ngh.neting.cc/argoproj/argo/cmd/argoexec/commands.NewWaitCommand.func1\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:16\ngh.neting.cc/spf13/cobra.(*Command).execute\n\t/go/src/github.com/spf13/cobra/command.go:766\ngh.neting.cc/spf13/cobra.(*Command).ExecuteC\n\t/go/src/github.com/spf13/cobra/command.go:852\ngh.neting.cc/spf13/cobra.(*Command).Execute\n\t/go/src/github.com/spf13/cobra/command.go:800\nmain.main\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:17\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:201\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1333"```
@Ark-kun
Copy link
Member Author

Ark-kun commented Oct 4, 2019

@jessesuen We'd really like this to be fixed in the ongoing release.

I'll try to find a fix.

@Ark-kun Ark-kun mentioned this issue Oct 7, 2019
Merged
@simster7 simster7 removed their assignment Oct 7, 2019
Ark-kun added a commit to Ark-kun/argo that referenced this issue Oct 8, 2019
@Ark-kun
Stop failing if artifact file exists, but empty
abdcf02 
Empty output artifacts are a valid and desired use case. (Think of `grep` or any other filtering program where one of the outputs can be empty.)
Argo failing on empty outputs was a pretty bad surprise for us.

Fixes argoproj#1642
The problem was introduced in https://github.com/argoproj/argo/pulls/1247
@Ark-kun
Copy link
Member Author

Ark-kun commented Oct 8, 2019

Here is the PR: #1653

@jackiemazy
Copy link

Hi, I am new to Argo and Kubernetes. Which version did this issue get resolved? I'm using argo v2.4.3, but still getting this error:

time="2020-02-15T14:11:09Z" level=info msg="Waiting on main container"
time="2020-02-15T14:11:11Z" level=info msg="main container started with container ID: 4668066a9246af05f0944ebdb249d245cbb260b092ca167b549b3755ed5cbc77"
time="2020-02-15T14:11:11Z" level=info msg="Starting annotations monitor"
time="2020-02-15T14:11:11Z" level=info msg="docker wait 4668066a9246af05f0944ebdb249d245cbb260b092ca167b549b3755ed5cbc77"
time="2020-02-15T14:11:11Z" level=info msg="Starting deadline monitor"
time="2020-02-15T14:11:11Z" level=info msg="Main container completed"
time="2020-02-15T14:11:11Z" level=info msg="No output parameters"
time="2020-02-15T14:11:11Z" level=info msg="Saving output artifacts"
time="2020-02-15T14:11:11Z" level=info msg="Annotations monitor stopped"
time="2020-02-15T14:11:11Z" level=info msg="Staging artifact: result"
time="2020-02-15T14:11:11Z" level=info msg="Copying /tmp/result.txt from container base image layer to /tmp/argo/outputs/artifacts/result.tgz"
time="2020-02-15T14:11:11Z" level=info msg="Archiving 4668066a9246af05f0944ebdb249d245cbb260b092ca167b549b3755ed5cbc77:/tmp/result.txt to /tmp/argo/outputs/artifacts/result.tgz"
time="2020-02-15T14:11:11Z" level=info msg="sh -c docker cp -a 4668066a9246af05f0944ebdb249d245cbb260b092ca167b549b3755ed5cbc77:/tmp/result.txt - | gzip > /tmp/argo/outputs/artifacts/result.tgz"
time="2020-02-15T14:11:11Z" level=warning msg="path /tmp/result.txt does not exist in archive /tmp/argo/outputs/artifacts/result.tgz"
time="2020-02-15T14:11:11Z" level=error msg="executor error: path /tmp/result.txt does not exist in archive /tmp/argo/outputs/artifacts/result.tgz\ngh.neting.cc/argoproj/argo/errors.New\n\t/go/src/github.com/argoproj/argo/errors/errors.go:49\ngh.neting.cc/argoproj/argo/errors.Errorf\n\t/go/src/github.com/argoproj/argo/errors/errors.go:55\ngh.neting.cc/argoproj/argo/workflow/executor/docker.(*DockerExecutor).CopyFile\n\t/go/src/github.com/argoproj/argo/workflow/executor/docker/docker.go:66\ngh.neting.cc/argoproj/argo/workflow/executor.(*WorkflowExecutor).stageArchiveFile\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:347\ngh.neting.cc/argoproj/argo/workflow/executor.(*WorkflowExecutor).saveArtifact\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:248\ngh.neting.cc/argoproj/argo/workflow/executor.(*WorkflowExecutor).SaveArtifacts\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:234\ngh.neting.cc/argoproj/argo/cmd/argoexec/commands.waitContainer\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:59\ngh.neting.cc/argoproj/argo/cmd/argoexec/commands.NewWaitCommand.func1\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:16\ngh.neting.cc/spf13/cobra.(*Command).execute\n\t/go/src/github.com/spf13/cobra/command.go:766\ngh.neting.cc/spf13/cobra.(*Command).ExecuteC\n\t/go/src/github.com/spf13/cobra/command.go:852\ngh.neting.cc/spf13/cobra.(*Command).Execute\n\t/go/src/github.com/spf13/cobra/command.go:800\nmain.main\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:17\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:201\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1333"
time="2020-02-15T14:11:11Z" level=info msg="Killing sidecars"
time="2020-02-15T14:11:11Z" level=info msg="Alloc=5883 TotalAlloc=11512 Sys=70078 NumGC=3 Goroutines=9"
time="2020-02-15T14:11:11Z" level=fatal msg="path /tmp/result.txt does not exist in archive /tmp/argo/outputs/artifacts/result.tgz\ngh.neting.cc/argoproj/argo/errors.New\n\t/go/src/github.com/argoproj/argo/errors/errors.go:49\ngh.neting.cc/argoproj/argo/errors.Errorf\n\t/go/src/github.com/argoproj/argo/errors/errors.go:55\ngh.neting.cc/argoproj/argo/workflow/executor/docker.(*DockerExecutor).CopyFile\n\t/go/src/github.com/argoproj/argo/workflow/executor/docker/docker.go:66\ngh.neting.cc/argoproj/argo/workflow/executor.(*WorkflowExecutor).stageArchiveFile\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:347\ngh.neting.cc/argoproj/argo/workflow/executor.(*WorkflowExecutor).saveArtifact\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:248\ngh.neting.cc/argoproj/argo/workflow/executor.(*WorkflowExecutor).SaveArtifacts\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:234\ngh.neting.cc/argoproj/argo/cmd/argoexec/commands.waitContainer\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:59\ngh.neting.cc/argoproj/argo/cmd/argoexec/commands.NewWaitCommand.func1\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:16\ngh.neting.cc/spf13/cobra.(*Command).execute\n\t/go/src/github.com/spf13/cobra/command.go:766\ngh.neting.cc/spf13/cobra.(*Command).ExecuteC\n\t/go/src/github.com/spf13/cobra/command.go:852\ngh.neting.cc/spf13/cobra.(*Command).Execute\n\t/go/src/github.com/spf13/cobra/command.go:800\nmain.main\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:17\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:201\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1333"

@Ark-kun
Copy link
Member Author

Ark-kun commented May 21, 2020

Which version did this issue get resolved?

2.5.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Stop failing if artifact file exists, but empty Ark-kun/argo
4 participants
@simster7 @Ark-kun @sarabala1979 @jackiemazy