Using youki and cri-o to run Kubernetes #584
Comments
|
The status of latest main branch.
|
|
here are reproduction steps for multi-node cri-o kubernetes cluster using LXD vm's. this can also be replicated using minikube but I use a manual configuration approach and am unaware of minikube ways to use cri-o instead of docker (docker-less setup). I have been playing around to integrate other runtimes to cri-o for use with kubernetes in different scenarios, like single-node clusters, different architectures (like arm) and different host underlying filesystems (zfs/lvm/xfs), and then trying to find the more canonical way to implement them by using templates (lxc, snap, aptitude, yaml for kubernetes finally). I hink this solution has 2 parts: part 2: make a cluster-wide solution to be implemented in orchestrators (aws/gce/kubernetes/openshift) as they all comply to the same standard. This part could be consisted of: 1: cluster-wide runtimeClass, maybe tags(selectors in kube-slang) to specify arch/os/distro(dinamically/statically built) relevant to kube yamls: apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: youki
handler: youkiapiVersion: v1
kind: Pod
metadata:
name: rustest
labels:
name: rust
spec:
runtimeClassName: youki
containers:
- name: rust
image: quay.io/gattytto/rst:latest
resources:
requests:
memory: "64Mi"
cpu: 1
limits:
memory: "128Mi"
cpu: 1 |
|
@Furisto you think the "EACCES" error may be the same case as the previous errors of a wrong symlink or reference to the entrypoints? |
|
@gattytto The container is successfully created, but when it comes to executing the specified executable the error happens. Looks like it's a permission issue but we are running as root, so that's strange. I do not entirely understand your setup but could it be that youki itself is running in a user namespace created by LXD and it is not actually root but trying to access files in the rootfs that are owned by root? Can you strace the execution? Maybe that gives more insight. Does the same problem occur with runc? |
this error is only happening with youki (not runc) and also the container. I will put a middleman bash script to add the strace command to the youki runtime execution and come back with more details on the error, thanks for pointing out the possible source of the error. you can see the current setup in the config generator and the lxd base template files I use to automate my setups, basically Cri-o engine using cgroupfs in "pod" cgroups2 namespace. |
|
IT WORKS! it turned out to be an error on my end, when using this docker image. I can not troubleshoot it in detail, I tried with nushell (a shell made with rust) but it didn't work. then I tried just using a basic example as starting point, it uses musl and it now works, at least for the most basic I am testing which is "wait" for 5 minutes, it gets restarted every time it finishes. now I see an error when trying to execute commands (apps) inside the container kubectl get pod
NAME READY STATUS RESTARTS AGE
rustest 1/1 Running 3 (2m55s ago) 18m
root@optimum-mayfly:~# kubectl exec pod/rustest -- /myip
Error: failed to load init spec
Caused by:
0: failed to load spec
1: io operation failed
2: No such file or directory (os error 2)
openat(AT_FDCWD, "/run/youki/183af8ddd214c67e446a4b0f1db7aefddba3a22d4656e2b847233f1ced3bfb27/config.json", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
write(2, "Error: ", 7Error: ) = 7
write(2, "failed to load init spec", 24failed to load init spec) = 24
root@driven-lizard:/var/log# ls /run/youki/183af8ddd214c67e446a4b0f1db7aefddba3a22d4656e2b847233f1ced3bfb27/
notify.sock state.json youki_config.json
|
|
FYI, I have just submitted a PR to add support into the Kubespray project . |
|
@utam0k the first steps for using youki with cri-o for kubernetes pods is succesfull. pod runs and executes app, then app(check ip) fails because it was expected (no internet connectivity given to it). more advanced tests may be made like /healthz endpoint and kubernetes api connectivity using the kube library (and the use of the kubernetes env vars provided by the runtime). |
|
@gattytto 💯 Thanks for your challenge! I'm sorry, but I don't know about kubernetes a lot. What will the test look like?
|
yes I will bring some more info here for everyone that wants to participate in growing the test pod image spec here. here is a rough diagram that depicts part of that structure: when the container is started, it is provided with network connectivity, and also given some environment variables: "env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"TERM=xterm",
"HOSTNAME=rustest",
"KUBERNETES_PORT=tcp://[1101:300:1:2::1]:443",
"KUBERNETES_PORT_443_TCP=tcp://[1101:300:1:2::1]:443",
"KUBERNETES_PORT_443_TCP_PROTO=tcp",
"KUBERNETES_PORT_443_TCP_PORT=443",
"KUBERNETES_PORT_443_TCP_ADDR=1101:300:1:2::1",
"KUBERNETES_SERVICE_HOST=1101:300:1:2::1",
"KUBERNETES_SERVICE_PORT=443",
"KUBERNETES_SERVICE_PORT_HTTPS=443",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
]additionally to that the container is provided with mounts, one of them very important containing a "secret" (preshared key style) that it will use to communicate with the api-server of kubernetes. this is the "secret" mount provided to the container: root@driven-lizard:~# ls /var/lib/kubelet/pods/fa884568-7222-4f36-8ebe-9dd17000444a/volumes/kubernetes.io~projected/kube-api-access-pj2dh
ca.crt namespace tokenUsing the previously provided stuff, the container should have enough data to make a connection to the kube-api server, this can be archieved using the kube "client" feature in the "kube" cargo library. Through this API the app can query/create/edit kubernetes datastore objects (be them standard kubernetes ones or custom objects created using CRD=custom resource definitions) and many more things (if its given the right to), like starting other pods or services, or querying the state of other pods or services. For the HEALTHZ endpoint, what the app should do is to set up a web service to respond with "ok" or whatever is required so the kubernetes api can be guessing the status of the app running inside the container. HERE I found some guide around it Another thing that should be tested is the hability of the app to write logs to both stdout and logfiles in some folder, so they can be later collected by log collection services present in kubernetes. this is the yaml spec of a pod set up using the pod yaml in the above message of this thread: root@optimum-mayfly:~# kubectl describe pod/rustest
Name: rustest
Namespace: default
Priority: 0
Node: driven-lizard/2001:----:----:----:----:----:----:1bda
Start Time: Wed, 12 Jan 2022 13:56:56 -0300
Labels: name=rust
Annotations: cni.projectcalico.org/containerID: 4d072e67434531635bbb993da9ed09086de2748d8b715de4ecde4b805400e69b
cni.projectcalico.org/podIP: 1100:200::3e:2364/128
cni.projectcalico.org/podIPs: 1100:200::3e:2364/128
Status: Running
IP: 1100:200::3e:2364
IPs:
IP: 1100:200::3e:2364
Containers:
rust:
Container ID: cri-o://ac0f62e65526fea7083af03d834450cf2f2c8792e6c40c041990c720a87a72bf
Image: quay.io/gattytto/rst:latest
Image ID: quay.io/gattytto/rst@sha256:9fe146aafa680be662b002554cee87ab7ef35bddfbe3a5b516ee457a215fb79a
Port: <none>
Host Port: <none>
State: Running
Started: Wed, 12 Jan 2022 13:57:18 -0300
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pj2dh (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-pj2dh:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m58s default-scheduler Successfully assigned default/rustest to driven-lizard
Normal Pulling 3m57s kubelet Pulling image "quay.io/gattytto/rst:latest"
Normal Pulled 3m36s kubelet Successfully pulled image "quay.io/gattytto/rst:latest" in 20.701607356s
Normal Created 3m36s kubelet Created container rust
Normal Started 3m36s kubelet Started container rustfor the last and most difficult part, here are some docs on kubernetes api unit tests, which may be useful to test the api communication part of the app using docker. |
@gattytto can you help me to support this use case, I'm not aware of the current status of the project |
will do! |
|
I will try to replicate this for youki, thanks for bringing it up @utam0k |
Goals
refs #78 (comment)
The text was updated successfully, but these errors were encountered: