Argo CLI error - FATA - configmaps "workflow-controller-configmap" not found

[fclesio]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

At the time that I start Argo Workflows server via $ argo server I get the error configmaps "workflow-controller-configmap" not found.

All other commands (e.g. argo list, argo auth token) is not working as well.

We have full access to our pods via kubectl

Related issues:

• Argo server should start without configmap #2285
• argo server configmap fatal 2.5.0RC7+ #2158

Version

v3.4.1

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

`$ kubectl version -o yaml`                                                                                                              


clientVersion:
  buildDate: "2022-08-23T17:36:43Z"
  compiler: gc
  gitCommit: a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2
  gitTreeState: clean
  gitVersion: v1.25.0
  goVersion: go1.19
  major: "1"
  minor: "25"
  platform: darwin/amd64
kustomizeVersion: v4.5.7
serverVersion:
  buildDate: "2022-08-16T09:24:51Z"
  compiler: gc
  gitCommit: e55564cf3a1384026a54920174977659c8c56a50
  gitTreeState: clean
  gitVersion: v1.22.12-gke.2300
  goVersion: go1.16.15b7
  major: "1"
  minor: "22"
  platform: linux/amd64

$ argo version

argo: v3.4.1
  BuildDate: 2022-10-01T15:51:47Z
  GitCommit: 0546fef0b096d84c9e3362d2b241614e743ebe97
  GitTreeState: clean
  GitTag: v3.4.1
  GoVersion: go1.18.6
  Compiler: gc
  Platform: darwin/amd64

$ argo server

INFO[2022-10-20T11:40:44.217Z] not enabling pprof debug endpoints
INFO[2022-10-20T11:40:44.219Z]                                               authModes="[client]" baseHRef=/ managedNamespace= namespace=default secure=true ssoNamespace=default
INFO[2022-10-20T11:40:44.219Z] Generating Self Signed TLS Certificates for Secure Mode
INFO[2022-10-20T11:40:44.220Z] SSO disabled
FATA[2022-10-20T11:40:44.361Z] configmaps "workflow-controller-configmap" not found

Logs from the workflow controller

INFO[2022-10-20T11:40:44.217Z] not enabling pprof debug endpoints
INFO[2022-10-20T11:40:44.219Z]                                               authModes="[client]" baseHRef=/ managedNamespace= namespace=default secure=true ssoNamespace=default
INFO[2022-10-20T11:40:44.219Z] Generating Self Signed TLS Certificates for Secure Mode
INFO[2022-10-20T11:40:44.220Z] SSO disabled
FATA[2022-10-20T11:40:44.361Z] configmaps "workflow-controller-configmap" not found

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

[terrytangyuan]

Check if this returns anything kubectl get configmap workflow-controller-configmap.

[fclesio]

@terrytangyuan Same message:

Error from server (NotFound): configmaps "workflow-controller-configmap" not found

[ognjen-it]

I had the same problem. Try with:
argo server --auth-mode server -n argo

[fclesio]

Thanks @ognjen-it, same error:

FATA[2022-10-26T10:13:24.008Z] configmaps "workflow-controller-configmap" not found 

[ognjen-it]

@fclesio
It's practically impossible...

Please, try the next commands and share the full logs that you got.

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: workflow-controller-configmap
  namespace: argo
data: {}
EOF

And then:

argo server --auth-mode server -n argo

[fclesio]

@ognjen-it Worked. Thanks a lot.

The issue was that the configmap was missing. Thanks!

[GergelyMolnar]

My post at #7649

For some reason, the server wants to use the default kubernetes namespace, try stating the server with the argo namespace specified:

$ argo server --auth-mode server -n argo