chore: Automatically set kube context in development container #246

ckadner · 2022-09-15T05:30:21Z

Motivation

When using the containerized development environment make develop to run FVT tests, one needs to configure access to a Kubernetes or OpenShift cluster from inside the container. Which has to be done for every make develop session. This can be tricky when cloud provider specific CLI tools are needed to connect and authenticate to a cluster.

Currently there is a short paragraph in the FVT README about how to export a minified kubeconfig file and create that inside the container. It is tedious to repeat those steps for each make develop session and depending on OS, shell environment, editors and possible text encoding issue it is also error prone.

Modifications

This PR proposes to automatically create the kubeconfig file in a local and git-ignored directory inside the local project and automatically mount it to the develop container. All the user then has to do is connect and authenticate to the cluster in the shell that will be running make develop.

Result

Kubernetes context is ready inside the development container.

# shell environment, outside the develop container has access to K8s cluster
[modelmesh-serving_ckadner]$ kubectl get pods

NAME                                        READY   STATUS    RESTARTS   AGE
pod/etcd                                    1/1     Running   0          17m
pod/minio                                   1/1     Running   0          17m
pod/modelmesh-controller-387aef25be-ftyqu   1/1     Running   0          17m

[modelmesh-serving_ckadner]$ make develop

./scripts/build_devimage.sh
Pulling dev image kserve/modelmesh-controller-develop:6be58b09c25833c1...
Building dev image kserve/modelmesh-controller-develop:6be58b09c25833c1...
Image kserve/modelmesh-controller-develop:6be58b09c25833c1 has 14 layers
Tagging dev image kserve/modelmesh-controller-develop:6be58b09c25833c1 as latest
./scripts/develop.sh
[root@17c121286549 workspace]# kubectl get pods
NAME                                        READY   STATUS    RESTARTS   AGE
pod/etcd                                    1/1     Running   0          18m
pod/minio                                   1/1     Running   0          18m
pod/modelmesh-controller-387aef25be-ftyqu   1/1     Running   0          18m
[root@17c121286549 workspace]#

/cc @njhill

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>

njhill · 2022-09-15T22:16:27Z

scripts/develop.sh

+# create local copy of a kube-config file
+mkdir -p .kube/
+kubectl config view --minify --flatten 2> /dev/null > .kube/config
+
 declare -a docker_run_args=(
  -v "${PWD}:/workspace"
  -v "${PWD}/.bash_history:/root/.bash_history"
+  -v "${PWD}/.kube/config:/root/.kube/config"


Thanks @ckadner this looks good in principle. But why does the kube context need to be saved separately? Why not just mount the user's .kube directory to the right place so that the existing context is used directly:

Suggested change

# create local copy of a kube-config file

mkdir -p .kube/

kubectl config view --minify --flatten 2> /dev/null > .kube/config

declare -a docker_run_args=(

-v "${PWD}:/workspace"

-v "${PWD}/.bash_history:/root/.bash_history"

-v "${PWD}/.kube/config:/root/.kube/config"

declare -a docker_run_args=(

-v "${PWD}:/workspace"

-v "${HOME}/.bash_history:/root/.bash_history"

-v "${HOME}/.kube:/root/.kube"

(I think the .bash_history mount should be fixed too)

Hi @njhill -- thanks for your review!

Why not just mount the user's .kube directory ... directly

Because different cloud providers use various CLI tools to connect to K8s and OCP clusters, some of which do not create ~/.kube/config files (i.e. IKS, OpenShift on IBM Cloud, OCP on Fyre). In each of those cases, though, you can create that minified kubeconfig file and it works well.

I think the .bash_history mount should be fixed too

My intention for the .bashhistory was to keep only those commands that are relevant to ModelMesh development, as opposed to all other user command history (mine spans a wide range of completely unrelated commands that I do not want to click through)

Because different cloud providers use various CLI tools to connect to K8s and OCP clusters, some of which do not create ~/.kube/config files (i.e. IKS, OpenShift on IBM Cloud, OCP on Fyre). In each of those cases, though, you can create that minified kubeconfig file and it works well.

@ckadner I could very well be wrong but are you sure about this? The non-kubectl CLI tool you're referring to here is oc right? That shares the same kubectl context/config (see here). This is only a client-side thing, it shouldn't matter what kind of Kubernetes cluster you're connecting to...

My intention for the .bashhistory was to keep only those commands that are relevant to ModelMesh development.

Ah ok now I understand the ${PWD} rationale for that, sounds reasonable, thanks!

Hi @njhill -- yes oc does not use/create a ~/.kube/config file (see below) and for IKS I am using the IKS CLI and a tool to switch between clusters by pointing to different kube-config files which the ibmcloud CLI stores under ${HOME}/.bluemix/plugins/container-service/clusters/... i.e.

KUBECONFIG=/Users/ckadner/.bluemix/plugins/container-service/clusters/ckadner-modelmesh-dev-cbu3tugd0r5j02te049g/kube-config-aaa00-ckadner-modelmesh-dev.yml

This is very useful when I am working on different clusters in different terminal shells at the same time, where I need different kube contexts for each of the terminals.

For OpenShift on IBM Cloud or OCP on Fyre, I see this behavior:

# login to cluster [modelmesh-serving]$ oc login --token=sha256~u_o6N****zznde_3rw --server=https://c104-e.us-south.containers.cloud.ibm.com:30276 Logged into "https://c104-e.us-south.containers.cloud.ibm.com:30276" as "IAM#ckadner@us.ibm.com" using the token provided. You have access to 67 projects, the list has been suppressed. You can list all projects with 'oc projects' Using project "default". # list some pods [modelmesh-serving]$ kubectl get pods -n kube-system | head NAME READY STATUS RESTARTS AGE ibm-file-plugin-684495896f-x6vsp 1/1 Running 0 14d ibm-keepalived-watcher-6zg84 1/1 Running 0 14d ibm-keepalived-watcher-9zqw9 1/1 Running 0 14d ibm-master-proxy-static-10.87.76.105 2/2 Running 2 28d ibm-master-proxy-static-10.87.76.117 2/2 Running 2 28d ibm-storage-metrics-agent-5c89ffc69f-hqd4h 1/1 Running 0 4d20h ibm-storage-watcher-77847bfb4c-xp5jn 1/1 Running 0 14d ibmcloud-block-storage-driver-n99lg 1/1 Running 0 14d ibmcloud-block-storage-driver-rjtrg 1/1 Running 0 14d # see there is no kube config file [modelmesh-serving]$ cat ~/.kube/config cat: /Users/ckadner/.kube/config: No such file or directory # use my .kube folder mount in develop.sh script [modelmesh-serving]$ cat scripts/develop.sh | grep kube -v "${HOME}/.kube:/root/.kube" # start the container, no kube context [modelmesh-serving]$ make develop ./scripts/build_devimage.sh Pulling dev image kserve/modelmesh-controller-develop:6be58b09c25833c1... Building dev image kserve/modelmesh-controller-develop:6be58b09c25833c1 [+] Building 1.0s (10/10) FINISHED ... Image kserve/modelmesh-controller-develop:6be58b09c25833c1 has 14 layers Tagging dev image kserve/modelmesh-controller-develop:6be58b09c25833c1 as latest ./scripts/develop.sh [root@72d9bf620910 workspace]# kubectl get pods error: Missing or incomplete configuration info. Please point to an existing, complete config file: 1. Via the command-line flag --kubeconfig 2. Via the KUBECONFIG environment variable 3. In your home directory as ~/.kube/config To view or setup config directly use the 'config' command. [root@72d9bf620910 workspace]#

njhill · 2022-09-20T19:56:17Z

@ckadner how about

-v "${KUBECONFIG:-${PWD}/.kube/config}:/root/.kube/config"

ckadner · 2022-09-20T21:07:41Z

@ckadner how about

-v "${KUBECONFIG:-${PWD}/.kube/config}:/root/.kube/config"

Hi @njhill -- I think you meant the user's ~/.kube/config like this?

  -v "${KUBECONFIG:-${HOME}/.kube/config}:/root/.kube/config"

Unfortunately also does not work since, for IKS at least, the kubeconfig files under ${HOME}/.bluemix/plugins/container-service/clusters/ have local absolute file references, which will not be present if only mounting the the one kubeconfig file without the whole ${HOME}/.bluemix/ folder also mounted:

[root@5f681a6f418d workspace]# kubectl get pods
error: unable to read certificate-authority /Users/ckadner/.bluemix/plugins/container-service/clusters/ckadner-modelmesh-dev-cbu3tugd0r5j02te049g/ca-aaa00-ckadner-modelmesh-dev.pem for ckadner-modelmesh-dev/cbu3tugd0r5j02te049g due to open /Users/ckadner/.bluemix/plugins/container-service/clusters/ckadner-modelmesh-dev-cbu3tugd0r5j02te049g/ca-aaa00-ckadner-modelmesh-dev.pem: no such file or directory

Does the minified kubeconfig approach in this PR not work for you? Or you don't like creating another local file? We could put that into the user's $TMP dir instead or tuck away all local developer related artifacts into a .dev folder, like .bash_history and develop_image_name ...

njhill · 2022-09-20T21:20:06Z

I think you meant the user's ~/.kube/config like this?

Yes, sorry, that's what I meant!

the kubeconfig files under ${HOME}/.bluemix/plugins/container-service/clusters/ have local absolute file references,

Ah ok, fair enough

Does the minified kubeconfig approach in this PR not work for you?

No it's fine, it just seemed cleaner to me to link to the outside config if possible. But I understand the complications now so this seems fine. I guess my only concern is that this would overwrite an existing .kube/config file if it was run in the user's home dir.

ckadner · 2022-09-20T22:08:53Z

I guess my only concern is that this would overwrite an existing .kube/config file if it was run in the user's home dir.

Right, but it should not break anything since it would rewrite it with the same kubeconfig file contents -- just "minified" :-)

We could instead save all local development artifacts in a .dev folder inside the users current folder, which we assume to be the local clone of modelmesh-serving. Artifacts like .bash_history and .develop_image_name and the now new .kube/config file. WDYT?

njhill · 2022-09-21T19:37:38Z

Right, but it should not break anything since it would rewrite it with the same kubeconfig file contents -- just "minified" :-)

What if the saved config comes from a different location (due to env var setting)?

We could instead save all local development artifacts in a .dev folder inside the users current folder, which we assume to be the local clone of modelmesh-serving. Artifacts like .bash_history and .develop_image_name and the now new .kube/config file.

That sounds like a good idea to me!

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>

ckadner · 2022-09-30T18:45:05Z

Right, but it should not break anything since it would rewrite it with the same kubeconfig file contents -- just "minified" :-)

What if the saved config comes from a different location (due to env var setting)?

We could instead save all local development artifacts in a .dev folder inside the users current folder, which we assume to be the local clone of modelmesh-serving. Artifacts like .bash_history and .develop_image_name and the now new .kube/config file.

That sounds like a good idea to me!

@njhill -- I moved the .bash_history and .kube_config file inside a .dev folder:

Move .bash_history and .kube/config files into .dev directory [3fbdf19]

And added a little bit of text to the developer doc:

Update developer.md [7677434]

Could you you do one more review of this PR? Thank you!

njhill

Thanks @ckadner!

kserve-oss-bot · 2022-09-30T19:52:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ckadner, njhill

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [njhill]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

njhill · 2022-09-30T19:52:43Z

/lgtm

chore: Automatically set kube context in make develop

fca9721

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>

kserve-oss-bot requested a review from njhill September 15, 2022 05:30

ckadner added the test testing related bugs and fixes label Sep 15, 2022

ckadner changed the title ~~chore: Automatically set kube context in make develop~~ chore: Automatically set kube context in development container Sep 15, 2022

njhill reviewed Sep 15, 2022

View reviewed changes

ckadner requested a review from njhill September 16, 2022 20:20

ckadner added the development label Sep 16, 2022

ckadner added 2 commits September 29, 2022 21:03

Move .bash_history and .kube/config files into .dev directory

3fbdf19

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>

Update developer.md

7677434

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>

njhill approved these changes Sep 30, 2022

View reviewed changes

kserve-oss-bot added the approved label Sep 30, 2022

kserve-oss-bot assigned njhill Sep 30, 2022

kserve-oss-bot added the lgtm label Sep 30, 2022

kserve-oss-bot merged commit 3cfb2db into kserve:main Sep 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Automatically set kube context in development container #246

chore: Automatically set kube context in development container #246

ckadner commented Sep 15, 2022 •

edited

Loading

njhill Sep 15, 2022

ckadner Sep 16, 2022 •

edited

Loading

njhill Sep 17, 2022

ckadner Sep 17, 2022 •

edited

Loading

njhill commented Sep 20, 2022

ckadner commented Sep 20, 2022

njhill commented Sep 20, 2022

ckadner commented Sep 20, 2022

njhill commented Sep 21, 2022

ckadner commented Sep 30, 2022

njhill left a comment

kserve-oss-bot commented Sep 30, 2022

njhill commented Sep 30, 2022

chore: Automatically set kube context in development container #246

chore: Automatically set kube context in development container #246

Conversation

ckadner commented Sep 15, 2022 • edited Loading

Motivation

Modifications

Result

njhill Sep 15, 2022

Choose a reason for hiding this comment

ckadner Sep 16, 2022 • edited Loading

Choose a reason for hiding this comment

njhill Sep 17, 2022

Choose a reason for hiding this comment

ckadner Sep 17, 2022 • edited Loading

Choose a reason for hiding this comment

njhill commented Sep 20, 2022

ckadner commented Sep 20, 2022

njhill commented Sep 20, 2022

ckadner commented Sep 20, 2022

njhill commented Sep 21, 2022

ckadner commented Sep 30, 2022

njhill left a comment

Choose a reason for hiding this comment

kserve-oss-bot commented Sep 30, 2022

njhill commented Sep 30, 2022

ckadner commented Sep 15, 2022 •

edited

Loading

ckadner Sep 16, 2022 •

edited

Loading

ckadner Sep 17, 2022 •

edited

Loading