Cannot run workflows with workflow templates

[alexec]

Summary

Error: workflowtemplates.argoproj.io "acollins8" not found

When trying to submit a workflow template (cluster install).

Diagnostics

What version of Argo Workflows are you running? latest

metadata:
  name: acollins8-6s8wf
  generateName: acollins8-
  namespace: default
  selfLink: /apis/argoproj.io/v1alpha1/namespaces/default/workflows/acollins8-6s8wf
  uid: 460de860-f423-44c1-bad0-e334637a4540
  resourceVersion: '33715'
  generation: 2
  creationTimestamp: '2020-08-13T20:33:38Z'
  labels:
    submit-from-ui: 'true'
    workflows.argoproj.io/completed: 'true'
    workflows.argoproj.io/phase: Error
    workflows.argoproj.io/resubmitted-from-workflow: acollins8-6nkft
    workflows.argoproj.io/workflow-template: acollins8
spec:
  entrypoint: argosay
  arguments:
    parameters:
      - name: message
        value: hello argo
  workflowTemplateRef:
    name: acollins8
status:
  phase: Error
  startedAt: '2020-08-13T20:33:38Z'
  finishedAt: '2020-08-13T20:33:38Z'
  message: workflowtemplates.argoproj.io "acollins8" not found
  storedTemplates:
    namespaced/acollins8/argosay:
      name: argosay
      arguments: {}
      inputs:
        parameters:
          - name: message
            value: '{{workflow.parameters.message}}'
      outputs: {}
      metadata: {}
      container:
        name: main
        image: 'argoproj/argosay:v2'
        command:
          - /argosay
        args:
          - echo
          - '{{inputs.parameters.message}}'
        resources: {}
  conditions:
    - type: Completed
      status: 'True'

argo-server-b6cd6579c-8h9jw workflow-controller-75dfb65464-bkggf workflow-controller time="2020-08-13T20:36:43Z" level=info msg="Processing workflow" namespace=default workflow=lovely-rhino-jf6q9
argo-serverworkflow-controller-75dfb65464-bkggf  time="2020-08-13T20:36:43Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=RetryWorkflow grpc.service=workflow.WorkflowService grpc.start_time="2020-08-13T20:36:43Z" grpc.time_ms=10.534 span.kind=server system=grpc
workflow-controller time="2020-08-13T20:36:43Z" level=error msg="Unable to get Workflow Template Reference for workflow" error="workflowtemplates.argoproj.io \"lovely-rhino\" not found" namespace=default workflow=lovely-rhino-jf6q9
workflow-controller-75dfb65464-bkggf workflow-controller time="2020-08-13T20:36:43Z" level=info msg="Updated phase Running -> Error" namespace=default workflow=lovely-rhino-jf6q9
workflow-controller-75dfb65464-bkggf workflow-controller time="2020-08-13T20:36:43Z" level=info msg="Updated message  -> workflowtemplates.argoproj.io \"lovely-rhino\" not found" namespace=default workflow=lovely-rhino-jf6q9
workflow-controller-75dfb65464-bkggf workflow-controller time="2020-08-13T20:36:43Z" level=info msg="Marking workflow completed" namespace=default workflow=lovely-rhino-jf6q9
workflow-controller-75dfb65464-bkggf workflow-controller time="2020-08-13T20:36:43Z" level=info msg="Checking daemoned children of " namespace=default workflow=lovely-rhino-jf6q9
workflow-controller-75dfb65464-bkggf workflow-controller time="2020-08-13T20:36:43Z" level=info msg="Workflow update successful" namespace=default phase=Error resourceVersion=34511 workflow=lovely-rhino-jf6q9
workflow-controller-75dfb65464-bkggf workflow-controller time="2020-08-13T20:36:44Z" level=info msg="Released all acquired locks" key=lovely-rhino-jf6q9
workflow-controller-75dfb65464-bkggf workflow-controller time="2020-08-13T20:36:44Z" level=info msg="Workflow update successful" namespace=default phase=Error resourceVersion=34511 workflow=lovely-rhino-jf6q9

Likely culprit: #3677 (#3680)

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[alexec]

Repro steps:

Do a cluster install with latest. Create a workflow template and then submit it. Failure.

[alexec]

Not a problem in 407f9e6.

[alexec]

Problem in da115f9 (and therefore due to it.