From 0c1da8e14aed154fb093b5bef303236a8c050565 Mon Sep 17 00:00:00 2001 From: Jeremy Lewi Date: Wed, 8 Apr 2020 08:18:32 -0700 Subject: [PATCH 1/7] Private GKE: Document image mirroring * Add instructions for mirroring docker images to private repositories * Fix kubeflow/kubeflow#3210 * Delete instructions under private GKE and just link to the doc issue #1705 * The instructions are outdated. Since managed certificates are used there should be no reason to need to update iap-ingress.yaml anymore. * Fix #1811 * Most of the other instructions under the private GKE section are also very obsolete. --- content/docs/gke/private-clusters.md | 173 ++++++++------------------- 1 file changed, 49 insertions(+), 124 deletions(-) diff --git a/content/docs/gke/private-clusters.md b/content/docs/gke/private-clusters.md index ee3a37027f..35d5aa7d6d 100644 --- a/content/docs/gke/private-clusters.md +++ b/content/docs/gke/private-clusters.md @@ -4,12 +4,12 @@ description = "How to secure Kubeflow clusters using VPC service controls and pr weight = 70 +++ -{{% alert title="Alpha version" color="warning" %}} +{{% alert title="Alpha" color="warning" %}} This feature is currently in **alpha** release status with limited support. The Kubeflow team is interested in any feedback you may have, in particular with regards to usability of the feature. Note the following issues already reported: -* [Documentation on how to use Kubeflow with shared VPC](https://github.com/kubeflow/kubeflow/issues/3082) +* [Documentation on how to use Kubeflow with private GKE and VPC service controls](https://github.com/kubeflow/website/issues/1705) * [Replicating Docker images to private Container Registry](https://github.com/kubeflow/kubeflow/issues/3210) * [Installing Istio for Kubeflow on private GKE](https://github.com/kubeflow/kubeflow/issues/3650) * [Profile-controller crashes on GKE private cluster](https://github.com/kubeflow/kubeflow/issues/4661) @@ -211,154 +211,79 @@ export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT} --format='value(proj --project=${PROJECT} ``` -## Deploy Kubeflow with Private GKE - -1. Set user credentials. You only need to run this command once: - - ``` - gcloud auth application-default login - ``` -1. Copy non-GCR hosted images to your GCR registry: - - 1. Clone the Kubeflow source - - ``` - git clone https://github.com/kubeflow/kubeflow.git git_kubeflow - ``` - 1. Use [Google Cloud Builder(GCB)](https://cloud.google.com/cloud-build/docs/) to replicate the images +## Mirror Kubeflow Application Images - ``` - cd git_kubeflow/scripts/gke - PROJECT= make copy-gcb - ``` - - * This is needed because your GKE nodes won't be able to pull images from non GCR - registries because they don't have public internet addresses - - - * gcloud may return an error even though the job is - submited successfully and will run successfully - see [kubeflow/kubeflow#3105](https://github.com/kubeflow/kubeflow/issues/3105) +Since private GKE can only access gcr.io, we need to mirror all images outside gcr.io for Kubeflow applications. We will use the `kfctl` tool to accomplish this. - * You can use the Cloud console to monitor your GCB job. - -1. Follow the guide to [deploying Kubeflow on GCP](/docs/gke/deploy/deploy-cli/). - When you reach the - [setup and deploy step](/docs/gke/deploy/deploy-cli/#set-up-and-deploy), - **skip the `kfctl apply` command** and run the **`kfctl build`** command - instead, as described in that step. Now you can edit the configuration files - before deploying Kubeflow. Retain the environment variables that you set - during the setup, including `${KF_NAME}`, `${KF_DIR}`, and `${CONFIG_FILE}`. - -1. Enable private clusters by editing `${KF_DIR}/gcp_config/cluster-kubeflow.yaml` and updating the following two parameters: - - ``` - privatecluster: true - gkeApiVersion: v1beta1 - ``` -1. Remove components which are not useful in private clusters: - - Open `${KF_DIR}/kfctl_gcp_iap.v1.0.0.yaml` and remove kustomizeConfig `cert-manager`, `cert-manager-crds`, and `cert-manager-kube-system-resources`. -1. Create the deployment: +1. Set your user credentials. You only need to run this command once: + ``` - cd ${KF_DIR} - kfctl apply -V -f ${CONFIG_FILE} + gcloud auth application-default login ``` - * If you get an error **legacy networks not supported**, follow the - [troubleshooting guide]( /docs/gke/troubleshooting-gke/#legacy-networks-are-not-supported) to create a new network. +1. Inside your `${KFAPP}` directory create a local configuration file `mirror.yaml` based on this [template](https://github.com/kubeflow/manifests/blob/master/experimental/mirror-images/gcp_template.yaml) - * You will need to manually create the network as a work around for [kubeflow/kubeflow#3071](https://github.com/kubeflow/kubeflow/issues/3071) + * Change destination to your project gcr registry. - ``` - cd ${KF_DIR}/gcp_config - gcloud --project=${PROJECT} deployment-manager deployments create ${KF_NAME}-network --config=network.yaml - ``` +1. Generate pipeline files to mirror images by running - * Then edit `${KF_DIR}/gcp_config/cluster.jinja` to add a field **network** in your cluster - - ``` - cluster: - name: {{ CLUSTER_NAME }} - network: - ``` - - * To get the name of the new network run - - ``` - gcloud --project=${PROJECT} compute networks list - ``` + ``` + cd ${KFAPP} + ./kfctl alpha mirror build mirror.yaml -V -o pipeline.yaml --gcb + ``` - * The name will contain the value ${KF_NAME} +1. Edit the couldbuild.yaml file -1. Update iap-ingress component parameters: + i. in the `images` section add - ``` - cd ${KF_DIR}/kustomize - gvim iap-ingress.yaml - ``` + ``` + - //docker.io/istio/proxy_init:1.1.6 + ``` - * Find and set the `privateGKECluster` parameter to true: + * Replace `/` with your registry - ``` - privateGKECluster: "true" - ``` + i. Under `steps` section addd - * Then apply your changes: + ``` + - args: + - build + - -t + - //docker.io/istio/proxy_init:1.1.6 + - --build-arg=INPUT_IMAGE=docker.io/istio/proxy_init:1.1.6 + - . + name: gcr.io/cloud-builders/docker + waitFor: + - '-' + ``` + i. Remove the mirroring of cos-nvidia-installer:fixed image. You don’t need it to be replicated because this image is privately available through GKE internal repo. - ``` - kubectl apply -f iap-ingress.yaml - ``` + 1. Remove the images from the `images` section + 1. Remove it from the steps section -1. Obtain an HTTPS certificate for your ${FQDN} and create a Kubernetes secret with it. - * You can create a self signed cert using [kube-rsa](https://github.com/kelseyhightower/kube-rsa) +1. Create a cloud build job to do the mirroring - ``` - go get github.com/kelseyhightower/kube-rsa - kube-rsa ${FQDN} - ``` - * The fully qualified domain is the host field specified for your ingress; - you can get it by running + ``` + gcloud builds submit --async gs://kubeflow-examples/image-replicate/replicate-context.tar.gz --project --config cloudbuild.yaml + ``` - ``` - cd ${KF_DIR}/kustomize - grep hostname: iap-ingress.yaml - ``` +1. Update your manifests to use the mirror'd images - * Then create your Kubernetes secret + ``` + kfctl alpha mirror overwrite -i pipeline.yaml + ``` - ``` - kubectl create secret tls --namespace=kubeflow envoy-ingress-tls --cert=ca.pem --key=ca-key.pem - ``` +1. Edit file “kustomize/istio-install/base/istio-noauth.yaml”: - * An alternative option is to upgrade to GKE 1.12 or later and use - [managed certificates](https://cloud.google.com/kubernetes-engine/docs/how-to/managed-certs#migrating_to_google-managed_certificates_from_self-managed_certificates) + 1. Replace “docker.io/istio/proxy_init:1.16” to gcr.io//docker.io/istio/proxy_init:1.16 + 1. Replace "docker.io/istio/proxyv2:1.1.6" to "gcr.io//docker.io/istio/proxyv2:1.1.6” - * See [kubeflow/kubeflow#3079](https://github.com/kubeflow/kubeflow/issues/3079) - -1. Update the various kustomize manifests to use `gcr.io` images instead of Docker Hub images. - -1. Apply all the Kubernetes resources: - - ``` - cd ${KF_DIR} - kfctl apply -V -f ${CONFIG_FILE} - ``` -1. Wait for Kubeflow to become accessible and then access it at this URL: - - ``` - https://${FQDN}/ - ``` - * ${FQDN} is the host associated with your ingress +## Deploy Kubeflow with Private GKE - * You can get it by running `kubectl get ingress` +{{% alert title="Coming Soon" color="warning" %}} +You can follow the issue: [Documentation on how to use Kubeflow with private GKE and VPC service controls](https://github.com/kubeflow/website/issues/1705) - * Follow the [instructions](/docs/gke/deploy/monitor-iap-setup/) to monitor the - deployment - - * It can take 10-20 minutes for the endpoint to become fully available ## Next steps From b214e4de92fe2d1f2f0a674ffd4783b8c2b11c2d Mon Sep 17 00:00:00 2001 From: Jeremy Lewi Date: Wed, 8 Apr 2020 08:49:04 -0700 Subject: [PATCH 2/7] Fix indentation. --- content/docs/gke/private-clusters.md | 89 ++++++++++++++-------------- 1 file changed, 46 insertions(+), 43 deletions(-) diff --git a/content/docs/gke/private-clusters.md b/content/docs/gke/private-clusters.md index 35d5aa7d6d..b043b054ab 100644 --- a/content/docs/gke/private-clusters.md +++ b/content/docs/gke/private-clusters.md @@ -158,61 +158,64 @@ export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT} --format='value(proj --add-access-levels=kubeflow \ --policy=${POLICYID} ``` -1. Set up container registry for GKE private clusters (for more info see [instructions](https://cloud.google.com/vpc-service-controls/docs/set-up-gke)): - - 1. Create a managed private zone - - ``` - export ZONE_NAME=kubeflow - export NETWORK= - gcloud beta dns managed-zones create ${ZONE_NAME} \ - --visibility=private \ - --networks=https://www.googleapis.com/compute/v1/projects/${PROJECT}/global/networks/${NETWORK} \ - --description="Kubeflow DNS" \ - --dns-name=gcr.io \ - --project=${PROJECT} - ``` - 1. Start a transaction +## Set up container registry for GKE private clusters (for more info see [instructions](https://cloud.google.com/vpc-service-controls/docs/set-up-gke)): - ``` - gcloud dns record-sets transaction start \ - --zone=${ZONE_NAME} \ - --project=${PROJECT} - ``` - 1. Add a CNAME record for \*.gcr.io +1. Create a managed private zone + + ``` + export ZONE_NAME=kubeflow + export NETWORK= + gcloud beta dns managed-zones create ${ZONE_NAME} \ + --visibility=private \ + --networks=https://www.googleapis.com/compute/v1/projects/${PROJECT}/global/networks/${NETWORK} \ + --description="Kubeflow DNS" \ + --dns-name=gcr.io \ + --project=${PROJECT} + ``` + +1. Start a transaction + + ``` + gcloud dns record-sets transaction start \ + --zone=${ZONE_NAME} \ + --project=${PROJECT} + ``` - ``` - gcloud dns record-sets transaction add \ - --name=*.gcr.io. \ - --type=CNAME gcr.io. \ + 1. Add a CNAME record for \*.gcr.io + + ``` + gcloud dns record-sets transaction add \ + --name=*.gcr.io. \ + --type=CNAME gcr.io. \ + --zone=${ZONE_NAME} \ + --ttl=300 \ + --project=${PROJECT} + ``` + + 1. Add an A record for the restricted VIP + + ``` + gcloud dns record-sets transaction add \ + --name=gcr.io. \ + --type=A 199.36.153.4 199.36.153.5 199.36.153.6 199.36.153.7 \ --zone=${ZONE_NAME} \ --ttl=300 \ --project=${PROJECT} - ``` - - 1. Add an A record for the restricted VIP - - ``` - gcloud dns record-sets transaction add \ - --name=gcr.io. \ - --type=A 199.36.153.4 199.36.153.5 199.36.153.6 199.36.153.7 \ - --zone=${ZONE_NAME} \ - --ttl=300 \ - --project=${PROJECT} - ``` + ``` - 1. Commit the transaction + 1. Commit the transaction - ``` - gcloud dns record-sets transaction execute \ - --zone=${ZONE_NAME} \ - --project=${PROJECT} - ``` + ``` + gcloud dns record-sets transaction execute \ + --zone=${ZONE_NAME} \ + --project=${PROJECT} + ``` ## Mirror Kubeflow Application Images + Since private GKE can only access gcr.io, we need to mirror all images outside gcr.io for Kubeflow applications. We will use the `kfctl` tool to accomplish this. From a29010d8f9257e4b91dc9739bf9983cfb032a95f Mon Sep 17 00:00:00 2001 From: Jeremy Lewi Date: Wed, 8 Apr 2020 08:53:21 -0700 Subject: [PATCH 3/7] Fix indetation. --- content/docs/gke/private-clusters.md | 56 ++++++++++++++-------------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/content/docs/gke/private-clusters.md b/content/docs/gke/private-clusters.md index b043b054ab..5c1d2cf6dd 100644 --- a/content/docs/gke/private-clusters.md +++ b/content/docs/gke/private-clusters.md @@ -227,43 +227,43 @@ Since private GKE can only access gcr.io, we need to mirror all images outside g 1. Inside your `${KFAPP}` directory create a local configuration file `mirror.yaml` based on this [template](https://github.com/kubeflow/manifests/blob/master/experimental/mirror-images/gcp_template.yaml) - * Change destination to your project gcr registry. + 1. Change destination to your project gcr registry. -1. Generate pipeline files to mirror images by running - - ``` - cd ${KFAPP} - ./kfctl alpha mirror build mirror.yaml -V -o pipeline.yaml --gcb - ``` +1. Generate pipeline files to mirror images by running + + ``` + cd ${KFAPP} + ./kfctl alpha mirror build mirror.yaml -V -o pipeline.yaml --gcb + ``` 1. Edit the couldbuild.yaml file - i. in the `images` section add + 1. in the `images` section add - ``` - - //docker.io/istio/proxy_init:1.1.6 - ``` - - * Replace `/` with your registry + 1. More + ``` + - //docker.io/istio/proxy_init:1.1.6 + ``` - i. Under `steps` section addd + * Replace `/` with your registry - ``` - - args: - - build - - -t - - //docker.io/istio/proxy_init:1.1.6 - - --build-arg=INPUT_IMAGE=docker.io/istio/proxy_init:1.1.6 - - . - name: gcr.io/cloud-builders/docker - waitFor: - - '-' - ``` - i. Remove the mirroring of cos-nvidia-installer:fixed image. You don’t need it to be replicated because this image is privately available through GKE internal repo. + 1. Under `steps` section add - 1. Remove the images from the `images` section - 1. Remove it from the steps section + ``` + - args: + - build + - -t + - //docker.io/istio/proxy_init:1.1.6 + - --build-arg=INPUT_IMAGE=docker.io/istio/proxy_init:1.1.6 + - . + name: gcr.io/cloud-builders/docker + waitFor: + - '-' + ``` + 1. Remove the mirroring of cos-nvidia-installer:fixed image. You don’t need it to be replicated because this image is privately available through GKE internal repo. + 1. Remove the images from the `images` section + 1. Remove it from the steps section 1. Create a cloud build job to do the mirroring From 517f45e0336c3b39041cc691a934b3695457f604 Mon Sep 17 00:00:00 2001 From: Jeremy Lewi Date: Wed, 8 Apr 2020 09:03:30 -0700 Subject: [PATCH 4/7] Fix indentation. --- content/docs/gke/private-clusters.md | 39 ++++++++++++++-------------- 1 file changed, 19 insertions(+), 20 deletions(-) diff --git a/content/docs/gke/private-clusters.md b/content/docs/gke/private-clusters.md index 5c1d2cf6dd..71dcc53425 100644 --- a/content/docs/gke/private-clusters.md +++ b/content/docs/gke/private-clusters.md @@ -215,7 +215,6 @@ export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT} --format='value(proj ## Mirror Kubeflow Application Images - Since private GKE can only access gcr.io, we need to mirror all images outside gcr.io for Kubeflow applications. We will use the `kfctl` tool to accomplish this. @@ -240,30 +239,30 @@ Since private GKE can only access gcr.io, we need to mirror all images outside g 1. in the `images` section add - 1. More - ``` - - //docker.io/istio/proxy_init:1.1.6 - ``` - - * Replace `/` with your registry + ``` + - //docker.io/istio/proxy_init:1.1.6 + ``` + + * Replace `/` with your registry 1. Under `steps` section add - ``` - - args: - - build - - -t - - //docker.io/istio/proxy_init:1.1.6 - - --build-arg=INPUT_IMAGE=docker.io/istio/proxy_init:1.1.6 - - . - name: gcr.io/cloud-builders/docker - waitFor: - - '-' - ``` + ``` + - args: + - build + - -t + - //docker.io/istio/proxy_init:1.1.6 + - --build-arg=INPUT_IMAGE=docker.io/istio/proxy_init:1.1.6 + - . + name: gcr.io/cloud-builders/docker + waitFor: + - '-' + ``` + 1. Remove the mirroring of cos-nvidia-installer:fixed image. You don’t need it to be replicated because this image is privately available through GKE internal repo. - 1. Remove the images from the `images` section - 1. Remove it from the steps section + 1. Remove the images from the `images` section + 1. Remove it from the steps section 1. Create a cloud build job to do the mirroring From cb5e6ea221516ccf1960cfbb874da8b47e5294e4 Mon Sep 17 00:00:00 2001 From: Jeremy Lewi Date: Wed, 8 Apr 2020 09:05:09 -0700 Subject: [PATCH 5/7] Fix alert. --- content/docs/gke/private-clusters.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/docs/gke/private-clusters.md b/content/docs/gke/private-clusters.md index 71dcc53425..db15d027b2 100644 --- a/content/docs/gke/private-clusters.md +++ b/content/docs/gke/private-clusters.md @@ -237,8 +237,8 @@ Since private GKE can only access gcr.io, we need to mirror all images outside g 1. Edit the couldbuild.yaml file - 1. in the `images` section add - + 1. In the `images` section add + ``` - //docker.io/istio/proxy_init:1.1.6 ``` @@ -285,7 +285,7 @@ Since private GKE can only access gcr.io, we need to mirror all images outside g {{% alert title="Coming Soon" color="warning" %}} You can follow the issue: [Documentation on how to use Kubeflow with private GKE and VPC service controls](https://github.com/kubeflow/website/issues/1705) - +{{% /alert %}} ## Next steps From 85e06f30d297bf93f557a9bd6c892ec20cbd4ec0 Mon Sep 17 00:00:00 2001 From: Jeremy Lewi Date: Wed, 8 Apr 2020 09:09:16 -0700 Subject: [PATCH 6/7] More formatting fixes. --- content/docs/gke/private-clusters.md | 56 ++++++++++++++-------------- 1 file changed, 29 insertions(+), 27 deletions(-) diff --git a/content/docs/gke/private-clusters.md b/content/docs/gke/private-clusters.md index db15d027b2..91c6c5e0dc 100644 --- a/content/docs/gke/private-clusters.md +++ b/content/docs/gke/private-clusters.md @@ -159,8 +159,10 @@ export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT} --format='value(proj --policy=${POLICYID} ``` -## Set up container registry for GKE private clusters (for more info see [instructions](https://cloud.google.com/vpc-service-controls/docs/set-up-gke)): +## Set up container registry for GKE private clusters: +Follow the step belows to configure your GCR registry to be accessible from your secured clusters. +For more info see [instructions](https://cloud.google.com/vpc-service-controls/docs/set-up-gke). 1. Create a managed private zone @@ -183,34 +185,34 @@ export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT} --format='value(proj --project=${PROJECT} ``` - 1. Add a CNAME record for \*.gcr.io +1. Add a CNAME record for \*.gcr.io - ``` - gcloud dns record-sets transaction add \ - --name=*.gcr.io. \ - --type=CNAME gcr.io. \ + ``` + gcloud dns record-sets transaction add \ + --name=*.gcr.io. \ + --type=CNAME gcr.io. \ + --zone=${ZONE_NAME} \ + --ttl=300 \ + --project=${PROJECT} + ``` + +1. Add an A record for the restricted VIP + + ``` + gcloud dns record-sets transaction add \ + --name=gcr.io. \ + --type=A 199.36.153.4 199.36.153.5 199.36.153.6 199.36.153.7 \ --zone=${ZONE_NAME} \ --ttl=300 \ --project=${PROJECT} - ``` - - 1. Add an A record for the restricted VIP - - ``` - gcloud dns record-sets transaction add \ - --name=gcr.io. \ - --type=A 199.36.153.4 199.36.153.5 199.36.153.6 199.36.153.7 \ - --zone=${ZONE_NAME} \ - --ttl=300 \ - --project=${PROJECT} - ``` + ``` - 1. Commit the transaction +1. Commit the transaction - ``` - gcloud dns record-sets transaction execute \ - --zone=${ZONE_NAME} \ - --project=${PROJECT} + ``` + gcloud dns record-sets transaction execute \ + --zone=${ZONE_NAME} \ + --project=${PROJECT} ``` ## Mirror Kubeflow Application Images @@ -238,7 +240,7 @@ Since private GKE can only access gcr.io, we need to mirror all images outside g 1. Edit the couldbuild.yaml file 1. In the `images` section add - + ``` - //docker.io/istio/proxy_init:1.1.6 ``` @@ -262,7 +264,7 @@ Since private GKE can only access gcr.io, we need to mirror all images outside g 1. Remove the mirroring of cos-nvidia-installer:fixed image. You don’t need it to be replicated because this image is privately available through GKE internal repo. 1. Remove the images from the `images` section - 1. Remove it from the steps section + 1. Remove it from the `steps` section 1. Create a cloud build job to do the mirroring @@ -278,8 +280,8 @@ Since private GKE can only access gcr.io, we need to mirror all images outside g 1. Edit file “kustomize/istio-install/base/istio-noauth.yaml”: - 1. Replace “docker.io/istio/proxy_init:1.16” to gcr.io//docker.io/istio/proxy_init:1.16 - 1. Replace "docker.io/istio/proxyv2:1.1.6" to "gcr.io//docker.io/istio/proxyv2:1.1.6” + 1. Replace `docker.io/istio/proxy_init:1.16` to `gcr.io//docker.io/istio/proxy_init:1.16` + 1. Replace `docker.io/istio/proxyv2:1.1.6` to `gcr.io//docker.io/istio/proxyv2:1.1.6` ## Deploy Kubeflow with Private GKE From ce6bd041a536338924b4860870174f4a143dd7bc Mon Sep 17 00:00:00 2001 From: Jeremy Lewi Date: Wed, 8 Apr 2020 09:29:27 -0700 Subject: [PATCH 7/7] Add comment about Tekton. --- content/docs/gke/private-clusters.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/content/docs/gke/private-clusters.md b/content/docs/gke/private-clusters.md index 91c6c5e0dc..0db6d40413 100644 --- a/content/docs/gke/private-clusters.md +++ b/content/docs/gke/private-clusters.md @@ -237,6 +237,9 @@ Since private GKE can only access gcr.io, we need to mirror all images outside g ./kfctl alpha mirror build mirror.yaml -V -o pipeline.yaml --gcb ``` + * If you want to use Tekton rather than Google Cloud Build(GCB) drop `--gcb` to emit a Tekton pipeline + * The instructions below assume you are using GCB + 1. Edit the couldbuild.yaml file 1. In the `images` section add