Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

istio-cni by default #2907

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

juliusvonkohout
Copy link
Member

@juliusvonkohout juliusvonkohout commented Nov 4, 2024

  • Istio-CNI as default
  • fix the ray tests
  • We should use the new "Values.gateways.seccompProfile" and securitycontext to get rid of the manual patch at
    - name: Configure istio init container with seccompProfile attribute
    run: |
    kubectl get cm istio-sidecar-injector -n istio-system -o yaml > temporary_patch.yaml
    sed -i '0,/runAsNonRoot: true/{s//&\n seccompProfile:\n type: RuntimeDefault/}' temporary_patch.yaml
    sed -i '/runAsNonRoot: true/{N; /runAsUser: {{ .ProxyUID | default "1337" }}/a\
    seccompProfile:\n type: RuntimeDefault
    }' temporary_patch.yaml
    kubectl apply -f temporary_patch.yaml
    rm temporary_patch.yaml
    both for istio and istio-cni, so it is related to Rootless Kubeflow #2528
  • We also need to add a comment about kserve with istio-cni
  • migrate https://github.com/kubeflow/manifests/blob/35298e042850e22a929ebdbcae94c225d69e33f8/.github/workflows/kserve_m2m_test.yaml to use a proper user namespace

@tarekabouzeid @kimwnasptd @DnPlas @kromanow94

Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from juliusvonkohout. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@juliusvonkohout
Copy link
Member Author

@hansinikarunarathne can you take a look at the ray tests?

@juliusvonkohout juliusvonkohout force-pushed the istio-cni-by-default-with-seccomp-value branch from 0bd1717 to 9f57cbc Compare November 19, 2024 17:11
@kromanow94
Copy link
Contributor

I asked @MaxKavun to check from our end. He should have some findings this week.

@MaxKavun
Copy link

MaxKavun commented Dec 9, 2024

I've done some checks

  • istio-init init container is gone (as per documentation)
  • istio-validation init container is added (as per documentation)
notebook:~/platform$ istioctl proxy-status
NAME                                                                           CLUSTER        CDS                LDS                EDS                RDS                ECDS        ISTIOD                      VERSION
activator-cdd86c7bc-7lqb4.knative-serving                                      Kubernetes     SYNCED (7m33s)     SYNCED (7m33s)     SYNCED (7m33s)     SYNCED (7m33s)     IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
autoscaler-5c8cf6fcb4-gs2w2.knative-serving                                    Kubernetes     SYNCED (27m)       SYNCED (27m)       SYNCED (27m)       SYNCED (27m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
cache-server-5c87d6f8f5-jg9pz.kubeflow                                         Kubernetes     SYNCED (24m)       SYNCED (24m)       SYNCED (24m)       SYNCED (24m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
centraldashboard-7d79c45f4d-pqm4j.kubeflow                                     Kubernetes     SYNCED (30m)       SYNCED (30m)       SYNCED (30m)       SYNCED (30m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
cluster-local-gateway-7d9dd5c5b6-qmswf.istio-system                            Kubernetes     SYNCED (19m)       SYNCED (19m)       SYNCED (19m)       SYNCED (19m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
controller-59fbd79b98-tll7s.knative-serving                                    Kubernetes     SYNCED (26m)       SYNCED (26m)       SYNCED (26m)       SYNCED (26m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
istio-ingressgateway-645cf78d8c-tv7tn.istio-system                             Kubernetes     SYNCED (15s)       SYNCED (14s)       SYNCED (15s)       SYNCED (14s)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
jupyter-web-app-deployment-67874f68d4-8s8df.kubeflow                           Kubernetes     SYNCED (21m)       SYNCED (21m)       SYNCED (21m)       SYNCED (21m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
katib-ui-657777bbff-b2cn8.kubeflow                                             Kubernetes     SYNCED (22m)       SYNCED (22m)       SYNCED (22m)       SYNCED (22m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
kserve-localmodel-controller-manager-94cf8bd66-5jrqv.kubeflow                  Kubernetes     SYNCED (7m13s)     SYNCED (7m13s)     SYNCED (7m13s)     SYNCED (7m13s)     IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
kserve-models-web-app-8485988f76-5bqrt.kubeflow                                Kubernetes     SYNCED (9m2s)      SYNCED (9m2s)      SYNCED (9m2s)      SYNCED (9m2s)      IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
metadata-grpc-deployment-d94cc8676-gkcx6.kubeflow                              Kubernetes     SYNCED (15m)       SYNCED (15m)       SYNCED (15m)       SYNCED (15m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
metadata-writer-6f4c6d898-m6t9z.kubeflow                                       Kubernetes     SYNCED (15m)       SYNCED (15m)       SYNCED (15m)       SYNCED (15m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
minio-5dc6ff5b96-wdqqz.kubeflow                                                Kubernetes     SYNCED (28m)       SYNCED (28m)       SYNCED (28m)       SYNCED (28m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
ml-pipeline-9c8bc4bc-wfv46.kubeflow                                            Kubernetes     SYNCED (9m26s)     SYNCED (9m26s)     SYNCED (9m26s)     SYNCED (9m26s)     IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
ml-pipeline-persistenceagent-5876c667f5-bmr9z.kubeflow                         Kubernetes     SYNCED (22m)       SYNCED (22m)       SYNCED (22m)       SYNCED (22m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
ml-pipeline-scheduledworkflow-57dcbc7799-8fwmm.kubeflow                        Kubernetes     SYNCED (12m)       SYNCED (12m)       SYNCED (12m)       SYNCED (12m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
ml-pipeline-ui-artifact-796cc85b9-gdjmz.kubeflow-user-example-com              Kubernetes     SYNCED (15m)       SYNCED (15m)       SYNCED (15m)       SYNCED (15m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
ml-pipeline-ui-dbc8ff8f8-g4ctq.kubeflow                                        Kubernetes     SYNCED (25m)       SYNCED (25m)       SYNCED (25m)       SYNCED (25m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
ml-pipeline-viewer-crd-75f665d644-w4wdh.kubeflow                               Kubernetes     SYNCED (9m59s)     SYNCED (9m59s)     SYNCED (9m59s)     SYNCED (9m59s)     IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
ml-pipeline-visualizationserver-5445b658d5-hjxz5.kubeflow                      Kubernetes     SYNCED (5m48s)     SYNCED (5m48s)     SYNCED (5m48s)     SYNCED (5m48s)     IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
ml-pipeline-visualizationserver-5c7579d99f-qvws4.kubeflow-user-example-com     Kubernetes     SYNCED (11m)       SYNCED (11m)       SYNCED (11m)       SYNCED (11m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
mysql-5b446b5744-mfbct.kubeflow                                                Kubernetes     SYNCED (9m25s)     SYNCED (9m25s)     SYNCED (9m25s)     SYNCED (9m25s)     IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
net-istio-webhook-d6684c5bc-wfqt7.knative-serving                              Kubernetes     SYNCED (13m)       SYNCED (13m)       SYNCED (13m)       SYNCED (13m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
notebook-controller-deployment-7978975c5b-xhr77.kubeflow                       Kubernetes     SYNCED (28m)       SYNCED (28m)       SYNCED (28m)       SYNCED (28m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
profiles-deployment-688cc95cdb-s2nnx.kubeflow                                  Kubernetes     SYNCED (7m44s)     SYNCED (7m44s)     SYNCED (7m44s)     SYNCED (7m44s)     IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
pvcviewer-controller-manager-5b8c866b58-wlww2.kubeflow                         Kubernetes     SYNCED (14m)       SYNCED (14m)       SYNCED (14m)       SYNCED (14m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
tensorboard-controller-deployment-85c9d7f7dd-f4sr8.kubeflow                    Kubernetes     SYNCED (10m)       SYNCED (10m)       SYNCED (10m)       SYNCED (10m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
tensorboards-web-app-deployment-777954f89d-f4n6n.kubeflow                      Kubernetes     SYNCED (12m)       SYNCED (12m)       SYNCED (12m)       SYNCED (12m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
volumes-web-app-deployment-7656678546-hs5dq.kubeflow                           Kubernetes     SYNCED (19m)       SYNCED (19m)       SYNCED (19m)       SYNCED (19m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
we-0.kubeflow-user-example-com                                                 Kubernetes     SYNCED (16m)       SYNCED (16m)       SYNCED (16m)       SYNCED (16m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
webhook-854c7ccc9-v9vq4.knative-serving                                        Kubernetes     SYNCED (12m)       SYNCED (12m)       SYNCED (12m)       SYNCED (12m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
workflow-controller-76bf47f998-qsdp2.kubeflow                                  Kubernetes     SYNCED (6m22s)     SYNCED (6m22s)     SYNCED (6m22s)     SYNCED (6m22s)     IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
notebook:~/platform$ istioctl analyze -A
Error [IST0145] (Gateway istio-system/cluster-local-gateway) Conflict with gateways knative-serving/knative-local-gateway (workload selector app=cluster-local-gateway,istio=cluster-local-gateway, port 80, hosts *).
Error [IST0145] (Gateway glo-k8s-admin/istio-ingressgateway) Conflict with gateways kubeflow/kubeflow-gateway (workload selector app=istio-ingressgateway,istio=ingressgateway, port 80, hosts *).
Error [IST0145] (Gateway knative-serving/knative-local-gateway) Conflict with gateways istio-system/cluster-local-gateway (workload selector app=cluster-local-gateway,istio=cluster-local-gateway, port 8081, hosts *).
Error [IST0145] (Gateway kubeflow/kubeflow-gateway) Conflict with gateways glo-k8s-admin/istio-ingressgateway (workload selector istio=ingressgateway, port 80, hosts *).
Warning [IST0133] (AuthorizationPolicy istio-system/istio-ingressgateway-require-jwt) Schema validation warning: configured AuthorizationPolicy will deny all traffic to TCP ports under its scope due to the use of only HTTP attributes in a DENY rule; it is recommended to explicitly specify the port

@juliusvonkohout juliusvonkohout force-pushed the istio-cni-by-default-with-seccomp-value branch from 9f57cbc to d676269 Compare December 9, 2024 11:45
@juliusvonkohout
Copy link
Member Author

I've done some checks

* istio-init init container is gone (as per documentation)

* istio-validation init container is added (as per documentation)
notebook:~/platform$ istioctl proxy-status
NAME                                                                           CLUSTER        CDS                LDS                EDS                RDS                ECDS        ISTIOD                      VERSION
activator-cdd86c7bc-7lqb4.knative-serving                                      Kubernetes     SYNCED (7m33s)     SYNCED (7m33s)     SYNCED (7m33s)     SYNCED (7m33s)     IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
autoscaler-5c8cf6fcb4-gs2w2.knative-serving                                    Kubernetes     SYNCED (27m)       SYNCED (27m)       SYNCED (27m)       SYNCED (27m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
cache-server-5c87d6f8f5-jg9pz.kubeflow                                         Kubernetes     SYNCED (24m)       SYNCED (24m)       SYNCED (24m)       SYNCED (24m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
centraldashboard-7d79c45f4d-pqm4j.kubeflow                                     Kubernetes     SYNCED (30m)       SYNCED (30m)       SYNCED (30m)       SYNCED (30m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
cluster-local-gateway-7d9dd5c5b6-qmswf.istio-system                            Kubernetes     SYNCED (19m)       SYNCED (19m)       SYNCED (19m)       SYNCED (19m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
controller-59fbd79b98-tll7s.knative-serving                                    Kubernetes     SYNCED (26m)       SYNCED (26m)       SYNCED (26m)       SYNCED (26m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
istio-ingressgateway-645cf78d8c-tv7tn.istio-system                             Kubernetes     SYNCED (15s)       SYNCED (14s)       SYNCED (15s)       SYNCED (14s)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
jupyter-web-app-deployment-67874f68d4-8s8df.kubeflow                           Kubernetes     SYNCED (21m)       SYNCED (21m)       SYNCED (21m)       SYNCED (21m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
katib-ui-657777bbff-b2cn8.kubeflow                                             Kubernetes     SYNCED (22m)       SYNCED (22m)       SYNCED (22m)       SYNCED (22m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
kserve-localmodel-controller-manager-94cf8bd66-5jrqv.kubeflow                  Kubernetes     SYNCED (7m13s)     SYNCED (7m13s)     SYNCED (7m13s)     SYNCED (7m13s)     IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
kserve-models-web-app-8485988f76-5bqrt.kubeflow                                Kubernetes     SYNCED (9m2s)      SYNCED (9m2s)      SYNCED (9m2s)      SYNCED (9m2s)      IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
metadata-grpc-deployment-d94cc8676-gkcx6.kubeflow                              Kubernetes     SYNCED (15m)       SYNCED (15m)       SYNCED (15m)       SYNCED (15m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
metadata-writer-6f4c6d898-m6t9z.kubeflow                                       Kubernetes     SYNCED (15m)       SYNCED (15m)       SYNCED (15m)       SYNCED (15m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
minio-5dc6ff5b96-wdqqz.kubeflow                                                Kubernetes     SYNCED (28m)       SYNCED (28m)       SYNCED (28m)       SYNCED (28m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
ml-pipeline-9c8bc4bc-wfv46.kubeflow                                            Kubernetes     SYNCED (9m26s)     SYNCED (9m26s)     SYNCED (9m26s)     SYNCED (9m26s)     IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
ml-pipeline-persistenceagent-5876c667f5-bmr9z.kubeflow                         Kubernetes     SYNCED (22m)       SYNCED (22m)       SYNCED (22m)       SYNCED (22m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
ml-pipeline-scheduledworkflow-57dcbc7799-8fwmm.kubeflow                        Kubernetes     SYNCED (12m)       SYNCED (12m)       SYNCED (12m)       SYNCED (12m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
ml-pipeline-ui-artifact-796cc85b9-gdjmz.kubeflow-user-example-com              Kubernetes     SYNCED (15m)       SYNCED (15m)       SYNCED (15m)       SYNCED (15m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
ml-pipeline-ui-dbc8ff8f8-g4ctq.kubeflow                                        Kubernetes     SYNCED (25m)       SYNCED (25m)       SYNCED (25m)       SYNCED (25m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
ml-pipeline-viewer-crd-75f665d644-w4wdh.kubeflow                               Kubernetes     SYNCED (9m59s)     SYNCED (9m59s)     SYNCED (9m59s)     SYNCED (9m59s)     IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
ml-pipeline-visualizationserver-5445b658d5-hjxz5.kubeflow                      Kubernetes     SYNCED (5m48s)     SYNCED (5m48s)     SYNCED (5m48s)     SYNCED (5m48s)     IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
ml-pipeline-visualizationserver-5c7579d99f-qvws4.kubeflow-user-example-com     Kubernetes     SYNCED (11m)       SYNCED (11m)       SYNCED (11m)       SYNCED (11m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
mysql-5b446b5744-mfbct.kubeflow                                                Kubernetes     SYNCED (9m25s)     SYNCED (9m25s)     SYNCED (9m25s)     SYNCED (9m25s)     IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
net-istio-webhook-d6684c5bc-wfqt7.knative-serving                              Kubernetes     SYNCED (13m)       SYNCED (13m)       SYNCED (13m)       SYNCED (13m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
notebook-controller-deployment-7978975c5b-xhr77.kubeflow                       Kubernetes     SYNCED (28m)       SYNCED (28m)       SYNCED (28m)       SYNCED (28m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
profiles-deployment-688cc95cdb-s2nnx.kubeflow                                  Kubernetes     SYNCED (7m44s)     SYNCED (7m44s)     SYNCED (7m44s)     SYNCED (7m44s)     IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
pvcviewer-controller-manager-5b8c866b58-wlww2.kubeflow                         Kubernetes     SYNCED (14m)       SYNCED (14m)       SYNCED (14m)       SYNCED (14m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
tensorboard-controller-deployment-85c9d7f7dd-f4sr8.kubeflow                    Kubernetes     SYNCED (10m)       SYNCED (10m)       SYNCED (10m)       SYNCED (10m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
tensorboards-web-app-deployment-777954f89d-f4n6n.kubeflow                      Kubernetes     SYNCED (12m)       SYNCED (12m)       SYNCED (12m)       SYNCED (12m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
volumes-web-app-deployment-7656678546-hs5dq.kubeflow                           Kubernetes     SYNCED (19m)       SYNCED (19m)       SYNCED (19m)       SYNCED (19m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
we-0.kubeflow-user-example-com                                                 Kubernetes     SYNCED (16m)       SYNCED (16m)       SYNCED (16m)       SYNCED (16m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
webhook-854c7ccc9-v9vq4.knative-serving                                        Kubernetes     SYNCED (12m)       SYNCED (12m)       SYNCED (12m)       SYNCED (12m)       IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
workflow-controller-76bf47f998-qsdp2.kubeflow                                  Kubernetes     SYNCED (6m22s)     SYNCED (6m22s)     SYNCED (6m22s)     SYNCED (6m22s)     IGNORED     istiod-69b6df4b56-2cd9d     1.23.2
notebook:~/platform$ istioctl analyze -A
Error [IST0145] (Gateway istio-system/cluster-local-gateway) Conflict with gateways knative-serving/knative-local-gateway (workload selector app=cluster-local-gateway,istio=cluster-local-gateway, port 80, hosts *).
Error [IST0145] (Gateway glo-k8s-admin/istio-ingressgateway) Conflict with gateways kubeflow/kubeflow-gateway (workload selector app=istio-ingressgateway,istio=ingressgateway, port 80, hosts *).
Error [IST0145] (Gateway knative-serving/knative-local-gateway) Conflict with gateways istio-system/cluster-local-gateway (workload selector app=cluster-local-gateway,istio=cluster-local-gateway, port 8081, hosts *).
Error [IST0145] (Gateway kubeflow/kubeflow-gateway) Conflict with gateways glo-k8s-admin/istio-ingressgateway (workload selector istio=ingressgateway, port 80, hosts *).
Warning [IST0133] (AuthorizationPolicy istio-system/istio-ingressgateway-require-jwt) Schema validation warning: configured AuthorizationPolicy will deny all traffic to TCP ports under its scope due to the use of only HTTP attributes in a DENY rule; it is recommended to explicitly specify the port

Hello @MaxKavun ,I rebased the PR. You can also create PRs against my branch to fix the remaining 3 items from the tasklist in the first post and the istioctl warnings.

@fraenkel
Copy link

The kuberay-operator needs to disable the init container injection as specified here:
https://docs.ray.io/en/latest/cluster/kubernetes/k8s-ecosystem/istio.html#step-3-optional-enable-istio-mtls-strict-mode

@juliusvonkohout
Copy link
Member Author

The kuberay-operator needs to disable the init container injection as specified here: https://docs.ray.io/en/latest/cluster/kubernetes/k8s-ecosystem/istio.html#step-3-optional-enable-istio-mtls-strict-mode

Thank you, do you mind creating a PR for this?

@juliusvonkohout
Copy link
Member Author

@MaxKavun @biswajit-9776 can you take a look at

juliusvonkohout and others added 3 commits December 18, 2024 09:15
Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com>
Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com>
When istio is enabled, the kuberay-operator must disable the init
container injection.

See https://docs.ray.io/en/latest/cluster/kubernetes/k8s-ecosystem/istio.html#step-3-optional-enable-istio-mtls-strict-mode

Signed-off-by: Michael Fraenkel <929377+fraenkel@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants