Update Knative for 1.4

[kimwnasptd]

Right now the manifests are providing the 0.17 version of Knative [source].

@kubeflow/wg-serving-leads @pvaneck which version of Knative would you like to have for the 1.4 Kubeflow release? IIRC we are planning to include the 0.6 version of KFServing, which has as a minimum requirement 0.17. Should we stick with this version for 1.4?

[pvaneck]

I believe we should do at least a minimum of Knative 0.19 (https://github.com/kubeflow/kfserving/tree/release-0.6#prerequisites) due to changes in the default local gateway (no longer using cluster-local-gateway - more info here).

[thunderboltsid]

Given current knative version is at 0.24 (and the operator supports only 3 versions back i.e. 0.21), what's the reason for keeping the knative dependency for Kubeflow to a version from last year? Are there some known issues with bundling the recent versions?

[pvaneck]

I personally don't mind using later Knative versions. Last release, Knative 0.17 was used in order to maintain compatibility with K8s 1.16 which some clusters were still using at the time. I'm not sure if there is a minimum recommended K8s version that is publicized for Kubeflow, but probably not worth supporting the way end-of-life versions which was what kept the Knative versions so far behind.

If Kubeflow Manifests is fine having a requirement of K8s 1.19+ for this release, then having Knative v0.24 is fine based on the Knative version table. However, I'm not sure if anyone has tested on v0.24 yet, as the KFServing e2e tests only test on v0.22, so that may be a better/easier candidate for this release.

[thunderboltsid]

If backward compatibility with older k8s versions is a concern (fair one at that), knative v0.23 seems to be compatible with k8s v1.18.x. Perhaps that can be a better candidate for the kubeflow v1.4 manifests. With that said, the k8s version skew policy states only the last three minor versions are supported at the moment, and patch support is provided up to 1y after release. K8s v1.18.x is already EOL, and v1.19.x will be EOL-ed end of September.

e2e testing is also an important concern. Do we know the amount of effort needed to test knative v0.23 on kfserving? If it's a relatively larger undertaking, perhaps we can go with v0.22 for now and plan v0.23/v0.24 for later releases.

[kimwnasptd]

I agree that we should update to Knative v0.22 for this release, since this is what KFServing is also using for their E2E tests.

Regarding the, future, upgrade to Knative 0.24 which has a K8s 1.19 as a minimum requirement we will need to also keep in mind the Pipelines' lineage with supporting K8s 1.19 out of the box. Adding some issues for that here for reference and context:
kubeflow/pipelines#5718
kubeflow/pipelines#5663 (comment)
kubeflow/pipelines#1654

cc @zijianjoy @Bobgy

[Bobgy]

Hi @kimwnasptd, there's nothing blocking people from upgrading to Kubernetes 1.19. The issues you mentioned -- KFP does not work by default on GKE 1.19, but I added documentation how they can configure container runtime to docker, refer to kubeflow/website#2857.

[kimwnasptd]

Ah, thanks for the clarification @Bobgy! Indeed, looking a little bit closer at the issues, it's for GKE 1.19 clusters which use containerd instead of docker by default for the created nodes.

[dprotaso]

FYI K8s 1.19 Is EOL/maintained mode sometime this week or next. Knative's next release will bump to K8s 1.20

Note Knative only tests N-1 to N upgrades - so big jumps are not really tested.