docker: invalid reference format

[Manvi07]

Steps to reproduce the issue:

1. curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.deb
2. sudo dpkg -i minikube_latest_amd64.deb
3. minikube start or
   minikube start --driver=docker or
   minikube start --driver=docker --base-image= gcr.io/k8s-minikube/kicbase:v0.0.10

Full output of failed command:

😄 minikube v1.10.1 on Ubuntu 18.04
✨ Using the docker driver based on existing profile
🆕 Kubernetes 1.18.2 is now available. If you would like to upgrade, specify: --kubernetes-version=1.18.2
🆕 Kubernetes 1.18.2 is now available. If you would like to upgrade, specify: --kubernetes-version=1.18.2
👍 Starting control plane node minikube in cluster minikube
🤷 docker "minikube" container is missing, will recreate.
E0518 19:32:57.136815 27695 oci.go:79] docker daemon seems to be stuck. Please try restarting your docker. Will try to delete anyways: unknown state "minikube": docker inspect minikube --format={{.State.Status}}: exit status 1
stdout:

stderr:
Template parsing error: template: :1:8: executing "" at <.State.Status>: map has no entry for key "State"
🔥 Creating docker container (CPUs=2, Memory=2200MB) ...
🤦 StartHost failed, but will try again: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --cpus=2 --memory=2200mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 : exit status 125
stdout:

stderr:
docker: invalid reference format.
See 'docker run --help'.

🤷 docker "minikube" container is missing, will recreate.
E0518 19:33:27.783172 27695 oci.go:79] docker daemon seems to be stuck. Please try restarting your docker. Will try to delete anyways: unknown state "minikube": docker inspect minikube --format={{.State.Status}}: exit status 1
stdout:

stderr:
Template parsing error: template: :1:8: executing "" at <.State.Status>: map has no entry for key "State"
🔥 Creating docker container (CPUs=2, Memory=2200MB) ...
😿 Failed to start docker container. "minikube start" may fix it: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --cpus=2 --memory=2200mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 : exit status 125
stdout:

stderr:
docker: invalid reference format.
See 'docker run --help'.

💣 error provisioning host: Failed to start host: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --cpus=2 --memory=2200mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 : exit status 125
stdout:

stderr:
docker: invalid reference format.
See 'docker run --help'.

😿 minikube is exiting due to an error. If the above message is not useful, open an issue:
👉 https://github.com/kubernetes/minikube/issues/new/choose

Failed attempts:

• I successfully installed minikube as described in the official minikube documentation. However, on starting the cluster it gave me the above error.

• I also killed all the docker containers, restarted docker using sudo systemctl start docker, and then tried the command minikube start again, but the error persists.

The traditional docker commands run without any errors.
(Tested using docker run hello-world)

Spec	Detail
Distro	Ubuntu 18.04.4 LTS
Docker Version	19.03.8
minikube version	v1.10.1

[afbjorklund]

Do you have an image called "minikube" ? Docker tends to get really confused if you do.

Unfortunately docker inspect doesn't know the difference between a container and an image.

If you run docker inspect minikube manually, it might show some clues ?

minikube is supposed to create a volume with the same name, but not an image

[Manvi07]

@afbjorklund thanks for replying.
I checked the list of images created by docker, there is none called "minikube". Also, minikube has created a volume and not an image as shown by docker inspect minikube.

[ { "CreatedAt": "2020-05-18T14:32:18+05:30", "Driver": "local", "Labels": { "created_by.minikube.sigs.k8s.io": "true", "name.minikube.sigs.k8s.io": "minikube" }, "Mountpoint": "/var/lib/docker/volumes/minikube/_data", "Name": "minikube", "Options": {}, "Scope": "local" } ]

[afbjorklund]

I think there is some other race condition at work, this issue talks about the same thing: #6795

Can't really reproduce it at the moment, but it seems to be going from volume straight to inspect...

And then both docker and podman are choking on trying to get the status of a non-existing container

It seems to be perfectly normal for it to react that way though, so that is why I asked about it:

 docker inspect foo
[]
Error: No such object: foo
 docker inspect foo --format={{.State.Status}}

Error: No such object: foo
 docker images busybox
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
busybox             glibc               845454170a51        4 days ago          5.2MB
busybox             latest              78096d0a5478        4 days ago          1.22MB
 docker inspect busybox --format={{.State.Status}}

Template parsing error: template: :1:8: executing "" at <.State.Status>: map has no entry for key "State"

But you are right that a volume could have the same failure scenario, probably what is happening

[afbjorklund]

I think that docker has a --type parameter, that could help here.

 docker inspect --type container foo
[]
Error: No such container: foo
 docker inspect --type image foo
[]
Error: No such image: foo

It is not available in podman, but it does have an exists command:

sudo podman container exists foo
sudo podman image exists foo

[Manvi07]

minikube works only with --type volume flag, giving the same JSON output as before.

[Manvi07]

Changing Kubernetes version to v1.12.10 solves the issue,

1. minikube delete
2. minikube start --kubernetes-version=v1.12.10

[afbjorklund]

minikube works only with --type volume flag, giving the same JSON output as before.

That was the whole point, that it should only look at the type intended.

It seems like the best fix is to change inspect to container inspect (etc)

[priyawadhwa]

Hey @Manvi07 are you still seeing this issue? I'm wondering if we just need to delete all containers and volumes named minikube and start again.

Does:

minikube delete
docker volume rm minikube
minikube start --driver docker

work?

[Manvi07]

Hey @Manvi07 are you still seeing this issue? I'm wondering if we just need to delete all containers and volumes named minikube and start again.

Does:

minikube delete
docker volume rm minikube
minikube start --driver docker

work?

Hi @priyawadhwa
Thanks for the suggestion. I did try this workaround back then but no luck!

[sharifelgamal]

Does upgrading to minikube v1.12.0 fix this issue for you @Manvi07 ?

[tstromberg]

This issue appears to be a duplicate of #8325, do you mind if we move the conversation there?

Ths way we can centralize the content relating to the issue. If you feel that this issue is not in fact a duplicate, please re-open it using /reopen. If you have additional information to share, please add it to the new issue.

Thank you for reporting this!