minikube 1.28.0 looking for image cache referencing k8s.gcr.io, should have been patched in 1.27?

[afewell]

What Happened?

I am running minikube 1.28.0 on ubuntu 20.04 and every time i stop and start minikube, it says Unable to load cached images: loading cached images: stat /home/viadmin/.minikube/cache/images/amd64/k8s.gcr.io/etcd_3.5.3-0: no such file or directory. It does eventually start, but seems to need to redownload the image and I need to optimize startup time.

I noticed in several previous issues including #14769 and pr's that k8s.gcr.io registry should have been changed to registry.k8s.io, and I understand that doesnt necessarily impact my local file path, but the path stated in the error message doesnt exist on my system, even when minikube is running, so either the cached image is not being created, or its looking in an incorrect location. My minikube logs show a large number of failed attempts to pull images from k8s.gcr.io but I dont see any reference to registry.k8s.io.

While minikube is running, docker ps shows that the running image is gcr.io/k8s-minikube/kicbase:v0.0.36. While my minikube cache does not have the file path shown in the error, it does appear to have a cache for the kic image:

viadmin@ubuntudesktop:~$ ll .minikube/cache/
total 16
drwxrwxr-x  4 viadmin docker  4096 Dec 19 05:49 ./
drwxr-xr-x 10 viadmin viadmin 4096 Jan  6 15:26 ../
drwxrwxr-x  3 viadmin docker  4096 Dec 19 05:49 kic/
drwxr-xr-x  2 viadmin docker  4096 Dec 19 05:49 preloaded-tarball/
viadmin@ubuntudesktop:~$ ll .minikube/cache/kic/
total 12
drwxrwxr-x 3 viadmin docker 4096 Dec 19 05:49 ./
drwxrwxr-x 4 viadmin docker 4096 Dec 19 05:49 ../
drwxrwxr-x 2 viadmin docker 4096 Dec 19 05:49 amd64/
viadmin@ubuntudesktop:~$ ll .minikube/cache/kic/amd64/
total 395556
drwxrwxr-x 2 viadmin docker      4096 Dec 19 05:49  ./
drwxrwxr-x 3 viadmin docker      4096 Dec 19 05:49  ../
-rw-rw-r-- 1 viadmin docker 405035520 Dec 19 05:49 'kicbase_v0.0.36@sha256_8debc1b6a335075c5f99bfbf131b4f5566f68c6500dc5991817832e55fcc9456.tar'

Attach the log file

log.txt

Operating System

Ubuntu

Driver

Docker

[afbjorklund]

It was changed in 1.24.9, while you pinned it to 1.24.8

• Changing the kubeadm registry was backported #15546

The main change was in k8s 1.25.0, not in k8s 1.27.0

• The DefaultKubernetesRepo changed for 1.25.0 #14768

---

The minikube releases are not related to the k8s releases

But all the needed images should be in the preloads ?

preloaded-images-k8s-v18-v1.24.8-docker-overlay2-amd64.tar.lz4

Maybe there was a bug generating it, to make it not match kubeadm.

[afbjorklund]

Seems OK.

https://dl.k8s.io/v1.24.8/bin/linux/amd64/kubeadm

$ ./kubeadm --kubernetes-version=v1.24.8 config images list
k8s.gcr.io/kube-apiserver:v1.24.8
k8s.gcr.io/kube-controller-manager:v1.24.8
k8s.gcr.io/kube-scheduler:v1.24.8
k8s.gcr.io/kube-proxy:v1.24.8
k8s.gcr.io/pause:3.7
k8s.gcr.io/etcd:3.5.5-0
k8s.gcr.io/coredns/coredns:v1.8.6

https://storage.googleapis.com/minikube-preloaded-volume-tarballs/v18/v1.24.8/preloaded-images-k8s-v18-v1.24.8-docker-overlay2-amd64.tar.lz4

docker@minikube:~$ docker images
REPOSITORY                                TAG       IMAGE ID       CREATED         SIZE
k8s.gcr.io/kube-apiserver                 v1.24.8   c7cbaca6e63b   8 weeks ago     130MB
k8s.gcr.io/kube-controller-manager        v1.24.8   9e2bfc195de6   8 weeks ago     119MB
k8s.gcr.io/kube-scheduler                 v1.24.8   9efa6dff568f   8 weeks ago     51MB
k8s.gcr.io/kube-proxy                     v1.24.8   a49578203a3c   8 weeks ago     110MB
k8s.gcr.io/etcd                           3.5.5-0   4694d02f8e61   3 months ago    300MB
k8s.gcr.io/pause                          3.7       221177c6082a   10 months ago   711kB
k8s.gcr.io/coredns/coredns                v1.8.6    a4ca41631cc7   15 months ago   46.8MB
gcr.io/k8s-minikube/storage-provisioner   v5        6e38f40d628d   21 months ago   31.5MB

[afbjorklund]

Normally the cache is not used for "docker" driver, unless you use --preload=false.

Only for those extra images that have been added locally, with minikube cache add.

[afewell]

@afbjorklund Ok thank you so much for helping me understand - it sounds like this is normal behavior and not a problem. I will close the issue ticket. thanks agin so much for helping me understand!

[afbjorklund]

No worries, hopefully it will all be merged for next minikube (1.29.0) ?

The end goal was to follow kubeadm, and to not have to redo preloads...

Changing the registry is somewhat artificial, when it comes to using preload.

But it does make a difference for the "none" driver, which doesn't use it.