Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot connect to the Docker daemon at unix:///var/run/docker.sock #1216

Closed
wardhane opened this issue Feb 13, 2019 · 5 comments
Closed

Cannot connect to the Docker daemon at unix:///var/run/docker.sock #1216

wardhane opened this issue Feb 13, 2019 · 5 comments

Comments

@wardhane
Copy link

Is this a BUG REPORT or FEATURE REQUEST?:
BUG REPORT
What happened:
I am running Argo on a kubernetes platform that is using containerd instead of Docker and the example https://github.com/argoproj/argo/blob/master/examples/artifact-passing.yaml fails with Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? message.

What you expected to happen:
Successful running of https://github.com/argoproj/argo/blob/master/examples/artifact-passing.yaml

How to reproduce it (as minimally and precisely as possible):

  1. Deploy Argo into a kubernetes cluster with containerd as the container runtime. Look at https://kubernetes.io/blog/2018/05/24/kubernetes-containerd-integration-goes-ga/ try it out section to know how to spin off a kubernetes cluster with containerd runtime
  2. Run argo submit --watch https://raw.githubusercontent.com/argoproj/argo/master/examples/artifact-passing.yaml
  3. Observe the call fail with the docker sock error shown above.

Anything else we need to know?:

Environment:

- Argo version:
argo: v2.2.1
  BuildDate: 2018-10-11T16:25:59Z
  GitCommit: 3b52b26190163d1f72f3aef1a39f9f291378dafb
  GitTreeState: clean
  GitTag: v2.2.1
  GoVersion: go1.10.3
  Compiler: gc
  Platform: darwin/amd64

$ argo version
  • Kubernetes version :
$ kubectl version -o yaml

clientVersion:
  buildDate: 2018-08-20T10:09:03Z
  compiler: gc
  gitCommit: 0c38c362511b20a098d7cd855f1314dad92c2780
  gitTreeState: clean
  gitVersion: v1.10.7
  goVersion: go1.9.3
  major: "1"
  minor: "10"
  platform: darwin/amd64
serverVersion:
  buildDate: 2019-02-01T10:26:11Z
  compiler: gc
  gitCommit: 4b76e71fd0eed7d9bda8f51ef5d61afe113d04ad
  gitTreeState: clean
  gitVersion: v1.12.5+IKS
  goVersion: go1.10.7
  major: "1"
  minor: "12"

Other debugging information (if applicable):

  • workflow result:
$ argo get <workflowname>

Name:                artifact-passing-9t2tg
Namespace:           argo
ServiceAccount:      workflow-sa
Status:              Failed
Message:             child 'artifact-passing-9t2tg-4053943796' failed
Created:             Wed Feb 13 15:47:26 +0000 (14 minutes ago)
Started:             Wed Feb 13 15:47:26 +0000 (14 minutes ago)
Finished:            Wed Feb 13 15:47:40 +0000 (14 minutes ago)
Duration:            14 seconds

STEP                       PODNAME                            DURATION  MESSAGE
 ✖ artifact-passing-9t2tg                                               child 'artifact-passing-9t2tg-4053943796' failed
 ├---✔ generate-artifact   artifact-passing-9t2tg-1311296340  8s
 └---✖ consume-artifact    artifact-passing-9t2tg-4053943796  5s        Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
  • executor logs:
$ kubectl logs <failedpodname> -c init
$ kubectl logs <failedpodname> -c wait

[harsha@AO-HWA-Mac my-bucket (⎈ |ptr040-temp:argo)]$ k logs artifact-passing-9t2tg-4053943796 -c wait
time="2019-02-13T15:47:39Z" level=info msg="Creating a docker executor"
time="2019-02-13T15:47:39Z" level=info msg="Executor (version: v2.2.1, build_date: 2018-10-11T16:27:29Z) initialized with template:\narchiveLocation:\n  s3:\n    accessKeySecret:\n      key: accesskey\n      name: argo-artifacts-minio\n    bucket: my-bucket\n    endpoint: argo-artifacts-minio.argo:9000\n    insecure: true\n    key: artifact-passing-9t2tg/artifact-passing-9t2tg-4053943796\n    secretKeySecret:\n      key: secretkey\n      name: argo-artifacts-minio\ncontainer:\n  args:\n  - cat /tmp/message\n  command:\n  - sh\n  - -c\n  image: alpine:latest\n  name: \"\"\n  resources: {}\ninputs:\n  artifacts:\n  - name: message\n    path: /tmp/message\n    s3:\n      accessKeySecret:\n        key: accesskey\n        name: argo-artifacts-minio\n      bucket: my-bucket\n      endpoint: argo-artifacts-minio.argo:9000\n      insecure: true\n      key: artifact-passing-9t2tg/artifact-passing-9t2tg-1311296340/hello-art.tgz\n      secretKeySecret:\n        key: secretkey\n        name: argo-artifacts-minio\nmetadata: {}\nname: print-message\noutputs: {}\n"
time="2019-02-13T15:47:39Z" level=info msg="Waiting on main container"
time="2019-02-13T15:47:40Z" level=info msg="main container started with container ID: e8d694c103e6d256d11ba68edf09b48e0b2649b4286d5d14d74e45ce17044f86"
time="2019-02-13T15:47:40Z" level=info msg="Starting annotations monitor"
time="2019-02-13T15:47:40Z" level=info msg="Starting deadline monitor"
time="2019-02-13T15:47:40Z" level=info msg="docker wait e8d694c103e6d256d11ba68edf09b48e0b2649b4286d5d14d74e45ce17044f86"
time="2019-02-13T15:47:40Z" level=error msg="`docker wait e8d694c103e6d256d11ba68edf09b48e0b2649b4286d5d14d74e45ce17044f86` failed: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n"
time="2019-02-13T15:47:40Z" level=info msg="Main container completed"
time="2019-02-13T15:47:40Z" level=info msg="No sidecars"
time="2019-02-13T15:47:40Z" level=info msg="No output artifacts"
time="2019-02-13T15:47:40Z" level=info msg="No output parameters"
time="2019-02-13T15:47:40Z" level=info msg="Annotations monitor stopped"
time="2019-02-13T15:47:40Z" level=info msg="Alloc=3357 TotalAlloc=11057 Sys=10342 NumGC=4 Goroutines=9"

$ kubectl logs -n kube-system $(kubectl get pods -l app=workflow-controller -n kube-system -o name)

time="2019-02-13T15:47:39Z" level=info msg="Workflow update successful" namespace=argo workflow=artifact-passing-9t2tg
time="2019-02-13T15:47:40Z" level=info msg="Processing workflow" namespace=argo workflow=artifact-passing-9t2tg
time="2019-02-13T15:47:40Z" level=info msg="Labeled pod argo/artifact-passing-9t2tg-1311296340 completed"
time="2019-02-13T15:47:40Z" level=info msg="Updating node artifact-passing-9t2tg[1].consume-artifact (artifact-passing-9t2tg-4053943796) status Running -> Failed"
time="2019-02-13T15:47:40Z" level=info msg="Updating node artifact-passing-9t2tg[1].consume-artifact (artifact-passing-9t2tg-4053943796) message: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
time="2019-02-13T15:47:40Z" level=info msg="SG Outbound nodes of artifact-passing-9t2tg-1311296340 are [artifact-passing-9t2tg-1311296340]" namespace=argo workflow=artifact-passing-9t2tg
time="2019-02-13T15:47:40Z" level=info msg="Step group node artifact-passing-9t2tg[1] (artifact-passing-9t2tg-4226338971) deemed failed: child 'artifact-passing-9t2tg-4053943796' failed" namespace=argo workflow=artifact-passing-9t2tg
time="2019-02-13T15:47:40Z" level=info msg="node artifact-passing-9t2tg[1] (artifact-passing-9t2tg-4226338971) phase Running -> Failed" namespace=argo workflow=artifact-passing-9t2tg
time="2019-02-13T15:47:40Z" level=info msg="node artifact-passing-9t2tg[1] (artifact-passing-9t2tg-4226338971) message: child 'artifact-passing-9t2tg-4053943796' failed" namespace=argo workflow=artifact-passing-9t2tg
time="2019-02-13T15:47:40Z" level=info msg="node artifact-passing-9t2tg[1] (artifact-passing-9t2tg-4226338971) finished: 2019-02-13 15:47:40.603274465 +0000 UTC" namespace=argo workflow=artifact-passing-9t2tg
time="2019-02-13T15:47:40Z" level=info msg="step group artifact-passing-9t2tg-4226338971 was unsuccessful: child 'artifact-passing-9t2tg-4053943796' failed" namespace=argo workflow=artifact-passing-9t2tg
time="2019-02-13T15:47:40Z" level=info msg="Outbound nodes of artifact-passing-9t2tg-4053943796 is [artifact-passing-9t2tg-4053943796]" namespace=argo workflow=artifact-passing-9t2tg
time="2019-02-13T15:47:40Z" level=info msg="Outbound nodes of artifact-passing-9t2tg is [artifact-passing-9t2tg-4053943796]" namespace=argo workflow=artifact-passing-9t2tg
time="2019-02-13T15:47:40Z" level=info msg="node artifact-passing-9t2tg (artifact-passing-9t2tg) phase Running -> Failed" namespace=argo workflow=artifact-passing-9t2tg
time="2019-02-13T15:47:40Z" level=info msg="node artifact-passing-9t2tg (artifact-passing-9t2tg) message: child 'artifact-passing-9t2tg-4053943796' failed" namespace=argo workflow=artifact-passing-9t2tg
time="2019-02-13T15:47:40Z" level=info msg="node artifact-passing-9t2tg (artifact-passing-9t2tg) finished: 2019-02-13 15:47:40.603645693 +0000 UTC" namespace=argo workflow=artifact-passing-9t2tg
time="2019-02-13T15:47:40Z" level=info msg="Checking deamoned children of artifact-passing-9t2tg" namespace=argo workflow=artifact-passing-9t2tg
time="2019-02-13T15:47:40Z" level=info msg="Updated phase Running -> Failed" namespace=argo workflow=artifact-passing-9t2tg
time="2019-02-13T15:47:40Z" level=info msg="Updated message  -> child 'artifact-passing-9t2tg-4053943796' failed" namespace=argo workflow=artifact-passing-9t2tg
time="2019-02-13T15:47:40Z" level=info msg="Marking workflow completed" namespace=argo workflow=artifact-passing-9t2tg
time="2019-02-13T15:47:40Z" level=info msg="Workflow update successful" namespace=argo workflow=artifact-passing-9t2tg
time="2019-02-13T15:47:41Z" level=info msg="Labeled pod argo/artifact-passing-9t2tg-4053943796 completed"
time="2019-02-13T15:47:41Z" level=info msg="Labeled pod argo/artifact-passing-9t2tg-1311296340 completed"
time="2019-02-13T15:47:56Z" level=info msg="Alloc=5284 TotalAlloc=55386 Sys=13514 NumGC=813 Goroutines=62"
time="2019-02-13T15:52:56Z" level=info msg="Alloc=4025 TotalAlloc=55433 Sys=13514 NumGC=815 Goroutines=62"
@w3irdrobot
Copy link

Argo attempts to mount the Docker socket from the host node into the wait container running in the workflow step pod. Since your instance of k8s doesn't have this because it's not using Docker, that is most likely the reason it is failing.

The code has a comment that explains the reasoning for this.

https://github.com/argoproj/argo/blob/f6b0c8f285217fd0e6089b0cf03ca4926d1b4758/workflow/controller/workflowpod.go#L48-L65

@wardhane
Copy link
Author

Thanks for replying. I guess kubectl cp works better in our case as the only guarantee is that kubernetes worker node will have a container runtime engine.

@rtheis
Copy link

rtheis commented Apr 1, 2019

Are there any plans to support other container runtimes, such as containerd?

@jessesuen
Copy link
Member

Yes, the PNS executor (v2.3) implemented in #1214 will support containerd.

Will close this as dupe of #970.

@asghaier76
Copy link

I am facing the same issue when deploying to IBM cloud. Is there a way to solve this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@w3irdrobot @jessesuen @asghaier76 @rtheis @wardhane