Update instructions for Kustomize 5 (kubeflow#2399)

* example: Update for kustomize 5

Add the sortOptions in the kustomization.yaml to ensure we are creating
MutationWebhooks before Deployments and Pods. This way we'll ensure that
Pods will get Istio sidecars in a race-free way.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* readme: Update instructions to use Kustomize 5

Update instructions to inform users to use Kustomize 5. We'll also need
to do an awk command, to remove a line that ends up in the final output.
That is a regression on Kustomize's side. Once a newer version of
Kustomize is available with the fix we'll then remove this awk command.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* gh-action: Update to Kustomize 5

Update the helper script to use Kustomize 5 for the tests.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* gh-action: Use the kustomize script in gh action

Update the GH Action that runs the unit test to use the same script we
use in the other GH Actions for fetching kustomize.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Add FAQ for earlier versions of Kustomize

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

---------

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

.github/workflows/manifests_unittests.yaml

@@ -13,12 +13,8 @@ jobs:
     - name: Check out repo
       uses: actions/checkout@v2
 
-    - name: Install Kustomize
-      working-directory: ./tests/e2e
-      run: |
-        curl -Lo ./kustomize https://github.com/kubernetes-sigs/kustomize/releases/download/v3.2.0/kustomize_3.2.0_linux_amd64
-        chmod +x ./kustomize
-        sudo mv kustomize /usr/local/bin
+    - name: Install kustomize
+      run: ./tests/gh-actions/install_kustomize.sh
 
     - name: Unit Test
       run: |

README.md

@@ -84,23 +84,25 @@ The `example` directory contains an example kustomization for the single command
 ### Prerequisites
 
 - `Kubernetes` (up to `1.25`) with a default [StorageClass](https://kubernetes.io/docs/concepts/storage/storage-classes/)
-- `kustomize` (version [`3.2.0`](https://github.com/kubernetes-sigs/kustomize/releases/tag/v3.2.0) or [`4.5.7`](https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv4.5.7))
-    - :warning: Kubeflow is compatible with Kustomize 4.5.7 only for [installing the individual components](#install-individual-components). The one-liner will need Kustomize 3.2, since versions from 4 and onwards have a regression on the order resources are sorted and printed. Please see [kubernetes-sigs/kustomize#3794](https://github.com/kubernetes-sigs/kustomize/issues/3794) and [kubeflow/manifests#1797](https://github.com/kubeflow/manifests/issues/1797). We know this is not ideal and are working with the upstream kustomize team to add support for the latest versions of kustomize as soon as we can.
+- `kustomize` [5.0.0](https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv5.0.0)
+    - :warning: Kubeflow is not compatible with earlier versions of Kustomize. This is because we need the [`sortOptions`](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/sortoptions/) field, which is only available in Kustomize 5 and onwards https://github.com/kubeflow/manifests/issues/2388.
 - `kubectl`
 
 ---
 **NOTE**
 
 `kubectl apply` commands may fail on the first try. This is inherent in how Kubernetes and `kubectl` work (e.g., CR must be created after CRD becomes ready). The solution is to simply re-run the command until it succeeds. For the single-line command, we have included a bash one-liner to retry the command.
 
+The reason we do `awk '!/well-defined/'` is because there's a regression in Kustomize 5 and a line is printed in stdout and not stderr https://github.com/kubernetes-sigs/kustomize/issues/5039. We'll remove this command once a future patch version of Kustomize is available.
+
 ---
 
 ### Install with a single command
 
 You can install all Kubeflow official components (residing under `apps`) and all common services (residing under `common`) using the following command:
 
 ```sh
-while ! kustomize build example | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done
+while ! kustomize build example | awk '!/well-defined/' | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done
 ```
 
 Once, everything is installed successfully, you can access the Kubeflow Central Dashboard [by logging in to your cluster](#connect-to-your-kubeflow-cluster).
@@ -242,7 +244,7 @@ kustomize build common/istio-1-16/kubeflow-istio-resources/base | kubectl apply
 Install the [Multi-User Kubeflow Pipelines](https://www.kubeflow.org/docs/components/pipelines/multi-user/) official Kubeflow component:
 
 ```sh
-kustomize build apps/pipeline/upstream/env/cert-manager/platform-agnostic-multi-user | kubectl apply -f -
+kustomize build apps/pipeline/upstream/env/cert-manager/platform-agnostic-multi-user | awk '!/well-defined/' | kubectl apply -f -
 ```
 This installs argo with the safe-to use runasnonroot emissary executor.  Please note that the installer is still responsible to analyze the security issues that arise when containers are run with root access and to decide if the kubeflow pipeline main containers are run as runasnonroot. It is strongly recommended that the pipelines main containers are installed and run as runasnonroot and without any special capabilities to mitigate security risks.
 
@@ -438,7 +440,5 @@ For security reasons, we don't want to use the default password for the default
 
 - **Q:** What versions of Istio, Knative, Cert-Manager, Argo, ... are compatible with Kubeflow? \
   **A:** Please refer to each individual component's documentation for a dependency compatibility range. For Istio, Knative, Dex, Cert-Manager and OIDC-AuthService, the versions in `common` are the ones we have validated.
-
-- **Q:** Can I use the latest Kustomize version (`v4.x`)? \
-  **A:** Kubeflow is compatible with Kustomize 4.5.7 only for [installing the individual components](#install-individual-components). The one-liner will need Kustomize 3.2, since versions from 4 and onwards have a regression on the order resources are sorted and printed. Please see [kubernetes-sigs/kustomize#3794](https://github.com/kubernetes-sigs/kustomize/issues/3794) and [kubeflow/manifests#1797](https://github.com/kubeflow/manifests/issues/1797). We know this is not ideal and are working with the upstream kustomize team to add support for the latest versions of kustomize as soon as we can.
-
+- **Q:** Can I use earlier version of Kustomize with Kubeflow manifests?
+  **A:** The manual installation instructions work with Kustomize 3.2. To use the one-liner installation you'll need to comment out the `sortOptions` section in the `example/kustomization.yaml`.

example/kustomization.yaml

@@ -1,6 +1,36 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 
+sortOptions:
+  order: legacy
+  legacySortOptions:
+    orderFirst:
+    - Namespace
+    - ResourceQuota
+    - StorageClass
+    - CustomResourceDefinition
+    - MutatingWebhookConfiguration
+    - ServiceAccount
+    - PodSecurityPolicy
+    - Role
+    - ClusterRole
+    - RoleBinding
+    - ClusterRoleBinding
+    - ConfigMap
+    - Secret
+    - Endpoints
+    - Service
+    - LimitRange
+    - PriorityClass
+    - PersistentVolume
+    - PersistentVolumeClaim
+    - Deployment
+    - StatefulSet
+    - CronJob
+    - PodDisruptionBudget
+    orderLast:
+    - ValidatingWebhookConfiguration
+
 resources:
 # Cert-Manager
 - ../common/cert-manager/cert-manager/base

tests/gh-actions/install_kustomize.sh

@@ -1,5 +1,6 @@
 #!/bin/bash
 set -e
-curl --silent --location --remote-name "https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize/v3.2.3/kustomize_kustomize.v3.2.3_linux_amd64"
-chmod a+x kustomize_kustomize.v3.2.3_linux_amd64
-sudo mv kustomize_kustomize.v3.2.3_linux_amd64 /usr/local/bin/kustomize
+curl --silent --location --remote-name "https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv5.0.0/kustomize_v5.0.0_linux_amd64.tar.gz"
+tar -xzvf kustomize_v5.0.0_linux_amd64.tar.gz
+chmod a+x kustomize
+sudo mv kustomize /usr/local/bin/kustomize