Creating workflows in sandbox projects return 500

[frsann]

I've set up Flyte sandbox using the instructions in the documentation. The set up works for the default projects flytesnacks and flytetester: registering the workflow and running them works fine.

I then create a new project

flyte-cli register-project --identifier flyteproj -n flyteporj -d "A cool ML proj to test Flyte" -h '<IP>:30081' -i

which works. I then register a workflow:

docker run --network host -e FLYTE_PLATFORM_URL='127.0.0.1:30081' myimage:<git hash> pyflyte -p flyteproj -d development -c sandbox.config register workflows

which also works.

However, when I launch the newly created workflow I only get a 500 error in the "Launch workflow" window.

The flyteadmin log looks like this:

{"json":{"src":"queues.go:76"},"level":"debug","msg":"refreshing execution queues","ts":"2020-01-24T11:24:20Z"}
{"json":{"src":"queues.go:116"},"level":"debug","msg":"refreshing workflow configs","ts":"2020-01-24T11:24:20Z"}
{"json":{"src":"queues.go:221"},"level":"debug","msg":"Evaluating execution queue for [{ResourceType:WORKFLOW Project:flyteproj Domain:development Name:single_step.edges.EdgeDetectorWf Version:772b42066c8a738ba3a06f165d131f3589a1533b XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] with available queues [[]] and available workflow configs [[]]","ts":"2020-01-24T11:24:20Z"}
{"json":{"src":"namespace_config_provider.go:34"},"level":"warning","msg":"Unsupported value for namespace_mapping in config, defaulting to \u003cproject\u003e-\u003cdomain\u003e","ts":"2020-01-24T11:24:20Z"}
{"json":{"src":"propeller_executor.go:130"},"level":"debug","msg":"failed to create workflow [resource_type:WORKFLOW project:\"flyteproj\" domain:\"development\" name:\"single_step.edges.EdgeDetectorWf\" version:\"772b42066c8a738ba3a06f165d131f3589a1533b\" [ in propeller namespaces \"flyteproj-development\" not found","ts":"2020-01-24T11:24:20Z"}
{"json":{"src":"execution_manager.go:259"},"level":"info","msg":"Failed to execute workflow {Project:flyteproj Domain:development Name: Spec:launch_plan:\u003cresource_type:LAUNCH_PLAN project:\"flyteproj\" domain:\"development\" name:\"single_step.edges.EdgeDetectorWf\" version:\"772b42066c8a738ba3a06f165d131f3589a1533b\" \u003e inputs:\u003cliterals:\u003ckey:\"image_input\" value:\u003cscalar:\u003cprimitive:\u003cstring_value:\"tttt\" \u003e \u003e \u003e \u003e \u003e metadata:\u003cprincipal:\"flyteconsole\" \u003e  Inputs:literals:\u003ckey:\"image_input\" value:\u003cscalar:\u003cprimitive:\u003cstring_value:\"tttt\" \u003e \u003e \u003e \u003e  XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} with execution id {Project:flyteproj Domain:development Name:rizu575do9 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} and inputs literals:\u003ckey:\"image_input\" value:\u003cscalar:\u003cprimitive:\u003cstring_value:\"tttt\" \u003e \u003e \u003e \u003e  with err failed to create workflow in propeller namespaces \"flyteproj-development\" not found","ts":"2020-01-24T11:24:20Z"}

Am I missing something?

Edit: Running on Minikube.

I noticed that syncresources is not running periodically. Shouldn't it?

[giordyb]

Something similar happened to me when the namespace somehow didn't get created.
Can you check if you have a namespace called "flyteproj-development" in your environment? if not can you try to create it manually and run the workflow again?

[frsann]

I actually got around the problem temporarily by running the syncresources job manually:

kubectl create job --from=cronjob/syncresources <job name> -n flyte

This seems to have created the required resources for my new project/workflows, and I could now execute new runs successfully.

The problem is probably not with Flyte but with with Kubernetes and with my cluster (Minikube) as CronJobs seem to have a difficulty recovering from restarting clusters, as reported here: kubernetes/kubernetes#42649.

[EngHabu]

If you run kubectl get cronjob -n flyte do you see this?

NAME            SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
syncresources   */1 * * * *   False     0        <none>          8s

I wonder what state does restarting the cluster leave the cron job in?

PR to apply one of the suggestions in the linked k8s issue: #172
Anybody willing to test it? (restart a minikube node... etc.)

[frsann]

@EngHabu
On a fresh Minikube cluster I get:

~ : kubectl get cronjob -n flyte                                                                                                                                                                                                                                  
NAME            SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
syncresources   */1 * * * *   False     1        6s              2m59s

and I can confirm that the syncresources job is running periodically.

~ : kubectl get pods -A
NAMESPACE              NAME                                         READY   STATUS              RESTARTS   AGE
<...>
flyte                  syncresources-1581683520-7kh8q               0/1     Completed           0          3m1s
flyte                  syncresources-1581683580-5kcx8               0/1     Completed           0          2m1s
flyte                  syncresources-1581683640-6f9ls               0/1     Completed           0          61s
flyte                  syncresources-1581683700-45zpl               0/1     ContainerCreating   0          1s
<...>

Now, creating a new project and launching an execution works as expected. This seems to support the hypothesis that the problem is related to cronjobs in minikube/kubernetes, and not Flyte per se.

EDIT: To clarify, the tests were carried out using unmodified configs, so not with the changes in the PR linked above.

[kumare3]

@frsann are you saying that you never tried the fix, but just restarting the cluster (and never suspend and resuming) it works as expected. But you have not tried the suggestion by @EngHabu ?. If you did try this would really help us resolve the issue

[frsann]

@kumare3 Correct. Fresh cluster without suspending+restarting worked for me. Also, running the cronjob manually fixes the symptom, but not the root cause.

I never tried the suggested changes.

[kumare3]

I think this is fixed @EngHabu / @katrogan any ideas?

[kumare3]

cc @mayitbeegh / @katrogan lets root cause this one and close it out? @katrogan can we just convert the cluster resource controller to be a golang daemon, instead of a cron job?

[EngHabu]

Fixed in https://github.com/flyteorg/flyte/releases/tag/v0.16.1