Upgrading Docker cluster from 1.9.x -> 1.10+: invalid reference format.

[balopat]

I had an old minikube instance created on 1.9.1 with the docker driver.
I did not delete it, just upgraded to 1.10.1 and ran minikube start --vm-driver=docker --container-runtime=docker

Got the following errors:

😄  minikube v1.10.1 on Darwin 10.15.5
✨  Using the docker driver based on existing profile
🆕  Kubernetes 1.18.2 is now available. If you would like to upgrade, specify: --kubernetes-version=1.18.2
🆕  Kubernetes 1.18.2 is now available. If you would like to upgrade, specify: --kubernetes-version=1.18.2
👍  Starting control plane node minikube in cluster minikube
💾  Downloading Kubernetes v1.17.3 preload ...
    > preloaded-images-k8s-v3-v1.17.3-docker-overlay2-amd64.tar.lz4: 522.49 MiB
🤷  docker "minikube" container is missing, will recreate.
E0529 14:59:32.318686   95883 oci.go:79] docker daemon seems to be stuck. Please try restarting your docker. Will try to delete anyways: unknown state "minikube": docker inspect minikube --format={{.State.Status}}: exit status 1
stdout:


stderr:
Error: No such object: minikube
🔥  Creating docker container (CPUs=2, Memory=5901MB) ...
🤦  StartHost failed, but will try again: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --cpus=2 --memory=5901mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 : exit status 125
stdout:

stderr:
docker: invalid reference format.
See 'docker run --help'.

🤷  docker "minikube" container is missing, will recreate.
E0529 15:00:03.939543   95883 oci.go:79] docker daemon seems to be stuck. Please try restarting your docker. Will try to delete anyways: unknown state "minikube": docker inspect minikube --format={{.State.Status}}: exit status 1
stdout:


stderr:
Template parsing error: template: :1:8: executing "" at <.State.Status>: map has no entry for key "State"
🔥  Creating docker container (CPUs=2, Memory=5901MB) ...
😿  Failed to start docker container. "minikube start" may fix it: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --cpus=2 --memory=5901mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 : exit status 125
stdout:

stderr:
docker: invalid reference format.
See 'docker run --help'.


💣  error provisioning host: Failed to start host: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --cpus=2 --memory=5901mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 : exit status 125
stdout:

stderr:
docker: invalid reference format.
See 'docker run --help'.


😿  minikube is exiting due to an error. If the above message is not useful, open an issue:
👉  https://github.com/kubernetes/minikube/issues/new/choose

After minikube delete things started to work

minikube start --vm-driver=docker --container-runtime=docker                                         391ms  Fri May 29 15:03:01 2020
😄  minikube v1.10.1 on Darwin 10.15.5
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.18.2 preload ...
    > preloaded-images-k8s-v3-v1.18.2-docker-overlay2-amd64.tar.lz4: 525.43 MiB
🔥  Creating docker container (CPUs=2, Memory=5901MB) ...
🐳  Preparing Kubernetes v1.18.2 on Docker 19.03.2 ...
    ▪ kubeadm.pod-network-cidr=10.244.0.0/16
🔎  Verifying Kubernetes components...
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"

❗  /Users/balintp/dev/tools/google-cloud-sdk/bin/kubectl is v1.14.10-dispatcher, which may be incompatible with Kubernetes v1.18.2.
💡  You can also use 'minikube kubectl -- get pods' to invoke a matching version

Docker version:

docker version                                                                                    1.4m  Fri May 29 15:04:26 2020
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:21:11 2020
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:29:16 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

[afbjorklund]

For some reason, the container image field got wiped when upgrading the config.

The error is from:

$ docker run ""
docker: invalid reference format.
See 'docker run --help'.

It was supposed to contain which image to use for kicbase, same as --base-image

[medyagh]

hm.... could it be that we can not upgrade if the base-image has changed ? (which we would need a container with a new base image rather than upgrade) ?

if thats the case, we still need to handle this better and let the user know we can not upgrade and they need to delete their existing cluster and create with new image.

it is also possible that we save the user's data which is a Volume, to be reused in the new image without loosing data

[medyagh]

@balopat I haevn't seen this issue again, I believe this has been fixed, I will close but please reopen if stil exists

[tstromberg]

I've seen two more reports of this issue, and noticed that KicBaseImage was recently added to the config. It does not appear that we assign a default value for configs which do not have KicBaseImage defined.

Looks like we need an upgrade integration test here.

[tstromberg]

This breakage occurs any time that we need to recreate the Docker container. Here is an easy reproduction case:

minikube-1.9.2 start --driver=docker
docker stop minikube
docker rm minikube
minikube-1.12.0 start --driver=docker

[tstromberg]

Fixed at head by #8738 - will be part of v1.12.1.