"Failed to establish pod watch: unknown" when following Getting Started w/ minikube

[crenshaw-dev]

Is this a BUG REPORT or FEATURE REQUEST?:

Bug report.

What happened:

I got failed to save outputs: Failed to establish pod watch: unknown (get pods) when running the hello-world workflow from the Getting Started page.

What you expected to happen:

I expected the hello-world workflow to complete successfully.

How to reproduce it (as minimally and precisely as possible):

minikube start
kubectl create namespace argo
kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo/stable/manifests/install.yaml
kubectl create rolebinding default-admin --clusterrole=admin --serviceaccount=argo:default
argo submit --watch https://raw.githubusercontent.com/argoproj/argo/master/examples/hello-world.yaml
# watch the workflow fail with the error mentioned above

I noticed the workflow seems to be running in namespace default. So I created a rolebinding with that namespace instead of argo

kubectl create rolebinding default-admin --clusterrole=admin --serviceaccount=default:default

Then this worked as expected:

argo submit --watch https://raw.githubusercontent.com/argoproj/argo/master/examples/hello-world.yaml

Anything else we need to know?:

I'm a k8s/Argo n00b, so I might have followed the guide wrong. But if not, I'm guessing the guide needs tweaking.

Environment:

• Argo version: v2.3.0

$ argo version
argo: v2.3.0
  BuildDate: 2019-05-20T22:11:09Z
  GitCommit: 88fcc70dcf6e60697e6716edc7464a403c49b27e
  GitTreeState: clean
  GitTag: v2.3.0
  GoVersion: go1.11.5
  Compiler: gc
  Platform: linux/amd64

• Kubernetes version :

$ kubectl version -o yaml
clientVersion:
  buildDate: "2019-10-07T14:30:40Z"
  compiler: gc
  gitCommit: d647ddbd755faf07169599a625faf302ffc34458
  gitTreeState: clean
  gitVersion: v1.16.1
  goVersion: go1.12.10
  major: "1"
  minor: "16"
  platform: linux/amd64
serverVersion:
  buildDate: "2019-09-18T14:27:17Z"
  compiler: gc
  gitCommit: 2bd9643cee5b3b3a5ecbd3af49d09018f0773c77
  gitTreeState: clean
  gitVersion: v1.16.0
  goVersion: go1.12.9
  major: "1"
  minor: "16"
  platform: linux/amd64

Other debugging information (if applicable):

• workflow result:

$ argo get hello-world-7rfh6
Name:                hello-world-7rfh6
Namespace:           default
ServiceAccount:      default
Status:              Error
Message:             failed to save outputs: Failed to establish pod watch: unknown (get pods)
Created:             Wed Oct 16 16:40:58 -0400 (14 minutes ago)
Started:             Wed Oct 16 16:40:58 -0400 (14 minutes ago)
Finished:            Wed Oct 16 16:41:19 -0400 (14 minutes ago)
Duration:            21 seconds

STEP                  PODNAME            DURATION  MESSAGE
 ⚠ hello-world-7rfh6  hello-world-7rfh6  21s       failed to save outputs: Failed to establish pod watch: unknown (get pods)

• executor logs:

$ kubectl logs hello-world-7rfh6 -c init
Error from server (BadRequest): container init is not valid for pod hello-world-7rfh6
$ kubectl logs hello-world-7rfh6 -c wait
time="2019-10-16T20:41:08Z" level=info msg="Creating a docker executor"
time="2019-10-16T20:41:08Z" level=info msg="Executor (version: v2.4.1, build_date: 2019-10-08T23:14:37Z) initialized (pod: default/hello-world-7rfh6) with template:\n{\"name\":\"whalesay\",\"arguments\":{},\"inputs\":{},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"docker/whalesay:latest\",\"command\":[\"cowsay\"],\"args\":[\"hello world\"],\"resources\":{}}}"
time="2019-10-16T20:41:08Z" level=info msg="Waiting on main container"
time="2019-10-16T20:41:08Z" level=error msg="executor error: Failed to establish pod watch: unknown (get pods)
time="2019-10-16T20:41:08Z" level=info msg="No output parameters"
time="2019-10-16T20:41:08Z" level=info msg="No output artifacts"
time="2019-10-16T20:41:08Z" level=info msg="Killing sidecars"
time="2019-10-16T20:41:08Z" level=warning msg="Failed to get pod 'hello-world-7rfh6': pods \"hello-world-7rfh6\" is forbidden: User \"system:serviceaccount:default:default\" cannot get resource \"pods\" in API group \"\" in the namespace \"default\""
time="2019-10-16T20:41:08Z" level=error msg="executor error: pods \"hello-world-7rfh6\" is forbidden: User \"system:serviceaccount:default:default\" cannot get resource \"pods\" in API group \"\" in the namespace \"default\"
time="2019-10-16T20:41:08Z" level=info msg="Alloc=4872 TotalAlloc=10015 Sys=70078 NumGC=3 Goroutines=7"
time="2019-10-16T20:41:08Z" level=fatal msg="pods \"hello-world-7rfh6\" is forbidden: User \"system:serviceaccount:default:default\" cannot get resource \"pods\" in API group \"\" in the namespace \"default\"

(I removed some seemingly-irrelevant noise from the logs. Let me know if this is insufficient.)

• workflow-controller logs:

$ kubectl logs -n argo $(kubectl get pods -l app=workflow-controller -n argo -o name)

These were long and full of stuff from unrelated tinkering. If they're necessary, let me know.

[ddseapy]

Looks like there was a recent change to install things in argo namespace instead of default:

https://github.com/argoproj/argo/pull/1656/files

Possibly some steps weren't updated?

You may want to try using the v2.3.0 docs if that's the version you are running. Maybe installation changed between the two versions.

[ddseapy]

Nevermind, I see that master points to v2.3.0. Yeah, looks like some instructions are still for the default namespace.

[crenshaw-dev]

@ddseapy do you think the instructions change needs to be reverted, or is there something else that needs to change to make the workflow use the argo namespace instead of default?

[simster7]

Thanks for the catch @mac9416. @ddseapy is correct -- we still intend to use the default namespace in this demo.