[marketplace] Kubeflow Pipelines doesn't work on Kubernetes 1.19 or above

[revathijay]

What steps did you take

Tried out the default kubeflow pipeline for TFX taxi prediction model in GCP marketplace Kubeflow pipeline version 1.4.1

What happened:

The error message for the csvexamplegen step is as follows.
This step is in Error state with this message: failed to save outputs: Error response from daemon: No such container: b715c336e667d305429003d1ce1c9b795ced595aa76b2bf252c679d2947b3643

What did you expect to happen:

The pipeline was getting executed end to end earlier . The main container itself for the first step is getting executed successfully. But its failing at the wait container stage.

Environment:

• How do you deploy Kubeflow Pipelines (KFP)?
• GCP AI pipeline V(1.4.1)

• KFP version:

• KFP SDK version:

Anything else you would like to add:

Labels

---

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[Ark-kun]

Do any other pipelines run successfully?

[carlosbertoncelli]

I had the same issue (with a custom pipeline), this started to happen after i launched a large number of runs and i had to stop the workloads from the GKE panel because the Kubeflow Pipelines dashboard wasnt responding.

Right now i cant run any kind of pipeline, not even the examples

[carlosbertoncelli]

I tried to reinstall the KFP using GCP Marketplace and didnt worked, i tried to delete the cluster and recreate everything from scratch and the KFP doesnt work anymore (i've also deleted the artifacts from GCP Bucket). Do you have any clues @Ark-kun ?

I've never experienced this issue before

[zijianjoy]

Hello @cabjr , Kubeflow Pipelines doesn't work on Kubernetes 1.19 or above, would you like to confirm the k8s version of your Kubeflow cluster? If it is the case, you might need to create a new cluster with K8s 1.18, and deploy Kubeflow pipelines on it.

Reference: #1654

[revathijay]

I finally got it working. it was because the creation of the cluster was blocked from the marketplace page. And I had to tweak the creation of the cluster manually with the rt set of options and the pipeline started working.