Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to write output artifacts when service account has limited permissions #1445

Closed
alexcapras opened this issue Jun 24, 2019 · 2 comments
Closed

Unable to write output artifacts when service account has limited permissions #1445

alexcapras opened this issue Jun 24, 2019 · 2 comments

Comments

@alexcapras
Copy link
Contributor

alexcapras commented Jun 24, 2019 •
edited
Loading

Is this a BUG REPORT or FEATURE REQUEST?:
FEATURE REQUEST

What happened:

I tried to run a workflow step that creates output artifacts. This runs fine if the service account being used to run the workflows has root permissions. However, if the account does not have permissions to write to the root directory /, the workflow step will fail with the below error message:

mkdir /argo/outputs: permission denied
github.com/argoproj/argo/errors.Wrap
    /root/go/src/github.com/argoproj/argo/errors/errors.go:87
github.com/argoproj/argo/errors.InternalWrapError
    /root/go/src/github.com/argoproj/argo/errors/errors.go:70
github.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).SaveArtifacts
    /root/go/src/github.com/argoproj/argo/workflow/executor/executor.go:203

What you expected to happen:

I expect the pod to write the output artifact, even if the service account being used does not have root permissions. I should not have to run as root in order to write output artifacts.

How to reproduce it (as minimally and precisely as possible):

Submit a workflow to argo, which has a step with an output artifact. Make sure the service account for the executor pod does not have permissions to write to the root directory. This can be done by adding the service account to spec.serviceAccount in the workflow yaml.

Anything else we need to know?:
No.

Environment:

  • Argo version:
argo: v2.2.1
  BuildDate: 2018-10-11T16:25:59Z
  GitCommit: 3b52b26190163d1f72f3aef1a39f9f291378dafb
  GitTreeState: clean
  GitTag: v2.2.1
  GoVersion: go1.10.3
  Compiler: gc
  Platform: darwin/amd64
  • Kubernetes version :
clientVersion:
  buildDate: 2018-11-26T14:38:32Z
  compiler: gc
  gitCommit: 637c7e288581ee40ab4ca210618a89a555b6e7e9
  gitTreeState: clean
  gitVersion: v1.10.11
  goVersion: go1.9.3
  major: "1"
  minor: "10"
  platform: darwin/amd64
serverVersion:
  buildDate: 2019-01-14T00:24:11Z
  compiler: gc
  gitCommit: d4cacc0
  gitTreeState: clean
  gitVersion: v1.11.0+d4cacc0
  goVersion: go1.10.3
  major: "1"
  minor: 11+
  platform: linux/amd64

Other debugging information (if applicable):

  • workflow result:
$ argo get <workflowname>
  • executor logs:
> oc logs run-setup-6-execution-dqdnx-135911213 -c init
time="2019-06-24T16:36:10Z" level=info msg="Creating a kubelet executor"
time="2019-06-24T16:36:10Z" level=info msg="Non configured envvar ARGO_KUBELET_PORT, defaulting the kubelet port to 10250"
time="2019-06-24T16:36:10Z" level=warning msg="Loading service account ca.crt as certificate authority to reach the kubelet api"
time="2019-06-24T16:36:10Z" level=info msg="Executor (version: v2.2.1, build_date: 2018-10-11T16:27:29Z) initialized with template:\narchiveLocation:\n  s3:\n    accessKeySecret:\n      key: accesskey\n      name: argo-artifacts-minio\n    bucket: imap-bucket\n    endpoint: imap-artifacts.argo-ns:9000\n    insecure: true\n    key: run-setup-6-execution-dqdnx/run-setup-6-execution-dqdnx-135911213\n    secretKeySecret:\n      key: secretkey\n      name: dev-imap-minio\ncontainer:\n  image: docker-registry.default.svc:5000/dev-imap/imap-model-wrapper:v1\n  name: \"\"\n  resources: {}\ninputs:\n  artifacts:\n  - name: model\n    path: /model.py\n    s3:\n      accessKeySecret:\n        key: accesskey\n        name: argo-artifacts-minio\n      bucket: imap-calc-models\n      endpoint: imap-minio.dev-imap:9000\n      insecure: true\n      key: model-58.py\n      secretKeySecret:\n        key: secretkey\n        name: argo-artifacts-minio\n  - name: data-contract-265\n    path: /tmp/inputs/data-source-265.csv\n    s3:\n      accessKeySecret:\n        key: accesskey\n        name: argo-artifacts-minio\n      bucket: imap-data-sources\n      endpoint: imap-minio.dev-imap:9000\n      insecure: true\n      key: data-source-156\n      secretKeySecret:\n        key: secretkey\n        name: argo-artifacts-minio\nmetadata:\n  annotations:\n    imapCalcModelId: \"58\"\nname: model-template-58\noutputs:\n  artifacts:\n  - name: data-contract-266\n    path: /tmp/outputs/data-source-266.csv\n"
time="2019-06-24T16:36:10Z" level=info msg="Start loading input artifacts..."
time="2019-06-24T16:36:10Z" level=info msg="Downloading artifact: model"
time="2019-06-24T16:36:10Z" level=info msg="Creating minio client imap-minio.dev-imap:9000 using static credentials"
time="2019-06-24T16:36:10Z" level=info msg="Getting from s3 (endpoint: imap-minio.dev-imap:9000, bucket: imap-calc-models, key: model-58.py) to /argo/inputs/artifacts/model.tmp"
time="2019-06-24T16:36:10Z" level=info msg="[tar -tf /argo/inputs/artifacts/model.tmp]"
time="2019-06-24T16:36:10Z" level=info msg="Successfully download file: /argo/inputs/artifacts/model"
time="2019-06-24T16:36:10Z" level=info msg="Downloading artifact: data-contract-265"
time="2019-06-24T16:36:10Z" level=info msg="Creating minio client imap-minio.dev-imap:9000 using static credentials"
time="2019-06-24T16:36:10Z" level=info msg="Getting from s3 (endpoint: imap-minio.dev-imap:9000, bucket: imap-data-sources, key: data-source-156) to /argo/inputs/artifacts/data-contract-265.tmp"
time="2019-06-24T16:36:10Z" level=info msg="[tar -tf /argo/inputs/artifacts/data-contract-265.tmp]"
time="2019-06-24T16:36:10Z" level=info msg="Successfully download file: /argo/inputs/artifacts/data-contract-265"
time="2019-06-24T16:36:10Z" level=info msg="Alloc=3409 TotalAlloc=10594 Sys=10598 NumGC=4 Goroutines=10"

> oc logs run-setup-6-execution-dqdnx-135911213 -c wait
time="2019-06-24T16:36:15Z" level=info msg="Creating a kubelet executor"
time="2019-06-24T16:36:15Z" level=info msg="Non configured envvar ARGO_KUBELET_PORT, defaulting the kubelet port to 10250"
time="2019-06-24T16:36:15Z" level=warning msg="Loading service account ca.crt as certificate authority to reach the kubelet api"
time="2019-06-24T16:36:15Z" level=info msg="Executor (version: v2.2.1, build_date: 2018-10-11T16:27:29Z) initialized with template:\narchiveLocation:\n  s3:\n    accessKeySecret:\n      key: accesskey\n      name: dev-imap-minio\n    bucket: imap-bucket\n    endpoint: imap-artifacts.dev-imap:9000\n    insecure: true\n    key: run-setup-6-execution-dqdnx/run-setup-6-execution-dqdnx-135911213\n    secretKeySecret:\n      key: secretkey\n      name: dev-imap-minio\ncontainer:\n  image: docker-registry.default.svc:5000/dev-imap/imap-model-wrapper:v1\n  name: \"\"\n  resources: {}\ninputs:\n  artifacts:\n  - name: model\n    path: /model.py\n    s3:\n      accessKeySecret:\n        key: accesskey\n        name: argo-artifacts-minio\n      bucket: imap-calc-models\n      endpoint: imap-minio.dev-imap:9000\n      insecure: true\n      key: model-58.py\n      secretKeySecret:\n        key: secretkey\n        name: argo-artifacts-minio\n  - name: data-contract-265\n    path: /tmp/inputs/data-source-265.csv\n    s3:\n      accessKeySecret:\n        key: accesskey\n        name: argo-artifacts-minio\n      bucket: imap-data-sources\n      endpoint: imap-minio.dev-imap:9000\n      insecure: true\n      key: data-source-156\n      secretKeySecret:\n        key: secretkey\n        name: argo-artifacts-minio\nmetadata:\n  annotations:\n    imapCalcModelId: \"58\"\nname: model-template-58\noutputs:\n  artifacts:\n  - name: data-contract-266\n    path: /tmp/outputs/data-source-266.csv\n"
time="2019-06-24T16:36:15Z" level=info msg="Waiting on main container"
time="2019-06-24T16:36:15Z" level=info msg="main container started with container ID: e0c0b7729f359c02a7824373d29b2666ac9655f3e1570e1b209c23661b3e205c"
time="2019-06-24T16:36:15Z" level=info msg="Starting annotations monitor"
time="2019-06-24T16:36:15Z" level=info msg="Starting to wait completion of containerID e0c0b7729f359c02a7824373d29b2666ac9655f3e1570e1b209c23661b3e205c ..."
time="2019-06-24T16:36:15Z" level=info msg="Starting deadline monitor"
time="2019-06-24T16:36:16Z" level=info msg="Main container completed"
time="2019-06-24T16:36:16Z" level=info msg="No sidecars"
time="2019-06-24T16:36:16Z" level=info msg="Saving output artifacts"
time="2019-06-24T16:36:16Z" level=info msg="Annotations monitor stopped"
time="2019-06-24T16:36:16Z" level=info msg="Saving artifact: data-contract-266"
time="2019-06-24T16:36:16Z" level=info msg="Archiving e0c0b7729f359c02a7824373d29b2666ac9655f3e1570e1b209c23661b3e205c:/tmp/outputs/data-source-266.csv to /argo/outputs/artifacts/data-contract-266.tgz"
time="2019-06-24T16:36:16Z" level=info msg="Alloc=4276 TotalAlloc=11333 Sys=10598 NumGC=4 Goroutines=9"
time="2019-06-24T16:36:16Z" level=fatal msg="unexpected non 200 status code: 403, body: Forbidden (user=system:serviceaccount:dev-imap:dev-imap-sa, verb=get, resource=nodes, subresource=proxy)\ngh.neting.cc/argoproj/argo/errors.Wrap\n\t/root/go/src/github.com/argoproj/argo/errors/errors.go:87\ngh.neting.cc/argoproj/argo/errors.InternalWrapError\n\t/root/go/src/github.com/argoproj/argo/errors/errors.go:70\ngh.neting.cc/argoproj/argo/workflow/executor/kubelet.(*kubeletClient).getCommandOutput\n\t/root/go/src/github.com/argoproj/argo/workflow/executor/kubelet/client.go:343\ngh.neting.cc/argoproj/argo/workflow/executor/kubelet.(*kubeletClient).CreateArchive\n\t/root/go/src/github.com/argoproj/argo/workflow/executor/kubelet/client.go:332\ngh.neting.cc/argoproj/argo/workflow/executor/kubelet.(*KubeletExecutor).CopyFile\n\t/root/go/src/github.com/argoproj/argo/workflow/executor/kubelet/kubelet.go:42\ngh.neting.cc/argoproj/argo/workflow/executor.(*WorkflowExecutor).saveArtifact\n\t/root/go/src/github.com/argoproj/argo/workflow/executor/executor.go:228\ngh.neting.cc/argoproj/argo/workflow/executor.(*WorkflowExecutor).SaveArtifacts\n\t/root/go/src/github.com/argoproj/argo/workflow/executor/executor.go:207\ngh.neting.cc/argoproj/argo/cmd/argoexec/commands.waitContainer\n\t/root/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:43\ngh.neting.cc/argoproj/argo/cmd/argoexec/commands.glob..func4\n\t/root/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:19\ngh.neting.cc/argoproj/argo/vendor/github.com/spf13/cobra.(*Command).execute\n\t/root/go/src/github.com/argoproj/argo/vendor/github.com/spf13/cobra/command.go:766\ngh.neting.cc/argoproj/argo/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/root/go/src/github.com/argoproj/argo/vendor/github.com/spf13/cobra/command.go:852\ngh.neting.cc/argoproj/argo/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/root/go/src/github.com/argoproj/argo/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/root/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:15\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:198\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:2361"
@alexcapras alexcapras changed the title Unable to write output artifact to MinIO when service account has limited permissions Unable to write output artifacts when service account has limited permissions Jul 1, 2019
@alexcapras
Copy link
Contributor Author

Looks like this can be easily fixed by changing the tempOutArtDir from "/argo/outputs/artifacts" to "/tmp/argo/outputs/artifacts"

https://github.com/argoproj/argo/blob/master/workflow/executor/executor.go#L48

@alexcapras alexcapras mentioned this issue Jul 2, 2019
Merged
@sarabala1979
Copy link
Member

Fixed in #1458

@nakfour nakfour mentioned this issue Mar 20, 2020
Closed
@Ark-kun Ark-kun mentioned this issue Apr 27, 2020
Closed
@LaVLaS LaVLaS mentioned this issue Jul 1, 2020
Closed
@juliusvonkohout juliusvonkohout mentioned this issue Oct 23, 2020
Closed
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@alexcapras @sarabala1979