minikube start -> panic: runtime error: invalid memory address or nil pointer dereference

[robrich]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Please provide the following details:

Environment:

Minikube version (use minikube version): minikube version: v0.33.0

• OS (e.g. from /etc/os-release): Windows 10 Pro 1809 (17763.253)
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): hyperv
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): file://C:/Users/.../.minikube/cache/iso/minikube-v0.33.0.iso
• Install tools: Downloaded minikube-windows-amd64 from github releases, renamed to minikube.exe and added to path
• Others: docker --version: Docker version 18.09.1, build 4c52b90
  The above can be generated in one go with the following commands (can be copied and pasted directly into your terminal):

minikube version
echo "";
echo "OS:";
cat /etc/os-release
echo "";
echo "VM driver:"; 
grep DriverName ~/.minikube/machines/minikube/config.json
echo "";
echo "ISO version";
grep -i ISO ~/.minikube/machines/minikube/config.json

What happened:

ran minikube.exe start --vm-driver="hyperv" --hyperv-virtual-switch="minikube" --alsologtostderr

got this:

Starting local Kubernetes v1.13.2 cluster...
Starting VM...
I0118 19:14:39.220301   12396 cluster.go:69] Machine does not exist... provisioning new machine
I0118 19:14:39.221086   12396 cluster.go:70] Provisioning machine with config: {MinikubeISO:https://storage.googleapis.com/minikube/iso/minikube-v0.33.0.iso Memory:2048 CPUs:2 DiskSize:20000 VMDriver:hyperv ContainerRuntime: HyperkitVpnKitSock: HyperkitVSockPorts:[] XhyveDiskDriver:ahci-hd DockerEnv:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch:minikube KvmNetwork:default Downloader:{} DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: GPU:false}
I0118 19:14:39.222153   12396 downloader.go:56] Not caching ISO, using https://storage.googleapis.com/minikube/iso/minikube-v0.33.0.iso
I0118 19:15:34.121149   12396 ssh_runner.go:101] SSH: sudo rm -f /etc/docker/ca.pem
I0118 19:15:34.175128   12396 ssh_runner.go:101] SSH: sudo mkdir -p /etc/docker
I0118 19:15:34.189696   12396 ssh_runner.go:101] SSH: sudo rm -f /etc/docker/server.pem
I0118 19:15:34.197665   12396 ssh_runner.go:101] SSH: sudo mkdir -p /etc/docker
I0118 19:15:34.213272   12396 ssh_runner.go:101] SSH: sudo rm -f /etc/docker/server-key.pem
I0118 19:15:34.219613   12396 ssh_runner.go:101] SSH: sudo mkdir -p /etc/docker
Getting VM IP address...
Moving files into cluster...
I0118 19:15:46.712967   12396 kubeadm.go:258] Container runtime flag provided with no value, using defaults.
I0118 19:15:46.712967   12396 ssh_runner.go:101] SSH: sudo rm -f /usr/bin/kubeadm
I0118 19:15:46.712967   12396 ssh_runner.go:101] SSH: sudo rm -f /usr/bin/kubelet
I0118 19:15:46.784255   12396 ssh_runner.go:101] SSH: sudo mkdir -p /usr/bin
I0118 19:15:46.785242   12396 ssh_runner.go:101] SSH: sudo mkdir -p /usr/bin
I0118 19:15:48.672998   12396 ssh_runner.go:101] SSH: sudo rm -f /lib/systemd/system/kubelet.service
I0118 19:15:48.680305   12396 ssh_runner.go:101] SSH: sudo mkdir -p /lib/systemd/system
I0118 19:15:48.694421   12396 ssh_runner.go:101] SSH: sudo rm -f /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
I0118 19:15:48.704130   12396 ssh_runner.go:101] SSH: sudo mkdir -p /etc/systemd/system/kubelet.service.d
I0118 19:15:48.721818   12396 ssh_runner.go:101] SSH: sudo rm -f /var/lib/kubeadm.yaml
I0118 19:15:48.728381   12396 ssh_runner.go:101] SSH: sudo mkdir -p /var/lib
I0118 19:15:48.744251   12396 ssh_runner.go:101] SSH: sudo rm -f /etc/kubernetes/addons/storage-provisioner.yaml
I0118 19:15:48.752317   12396 ssh_runner.go:101] SSH: sudo mkdir -p /etc/kubernetes/addons
I0118 19:15:48.768638   12396 ssh_runner.go:101] SSH: sudo rm -f /etc/kubernetes/manifests/addon-manager.yaml
I0118 19:15:48.777557   12396 ssh_runner.go:101] SSH: sudo mkdir -p /etc/kubernetes/manifests/
I0118 19:15:48.799286   12396 ssh_runner.go:101] SSH: sudo rm -f /etc/kubernetes/addons/storageclass.yaml
I0118 19:15:48.807055   12396 ssh_runner.go:101] SSH: sudo mkdir -p /etc/kubernetes/addons
I0118 19:15:48.823072   12396 ssh_runner.go:101] SSH: sudo rm -f /etc/kubernetes/addons/kube-dns-controller.yaml
I0118 19:15:48.831707   12396 ssh_runner.go:101] SSH: sudo mkdir -p /etc/kubernetes/addons
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x18 pc=0xfc6121]

goroutine 567 [running]:
k8s.io/minikube/pkg/minikube/assets.(*BinDataAsset).Read(0xc000315ce0, 0xc000636000, 0x8000, 0x8000, 0x8000, 0x8000, 0xc000473e58)
        /go/src/k8s.io/minikube/pkg/minikube/assets/vm_assets.go:168 +0x31
io.copyBuffer(0x3390138, 0xc0001756a0, 0x2bf6898, 0xc000315ce0, 0xc000636000, 0x8000, 0x8000, 0x12cf860, 0xc0001b0000, 0x2bf6898)
        /usr/local/go/src/io/io.go:402 +0x12c
io.Copy(0x3390138, 0xc0001756a0, 0x2bf6898, 0xc000315ce0, 0xc000315ce0, 0x21, 0x0)
        /usr/local/go/src/io/io.go:364 +0x61
k8s.io/minikube/pkg/minikube/bootstrapper.(*SSHRunner).Copy.func1(0xc00030f690, 0x1540b80, 0xc0001756a0, 0x155b6a0, 0xc000315ce0)
        /go/src/k8s.io/minikube/pkg/minikube/bootstrapper/ssh_runner.go:181 +0x34e
created by k8s.io/minikube/pkg/minikube/bootstrapper.(*SSHRunner).Copy
        /go/src/k8s.io/minikube/pkg/minikube/bootstrapper/ssh_runner.go:176 +0x418

Then ran this: minikube status

got this:

E0118 19:19:51.383798    1620 status.go:88] Error kubelet status: getting status: Process exited with status 3

What you expected to happen: Minikube to startup correctly

How to reproduce it (as minimally and precisely as possible):

Output of minikube logs (if applicable):

Anything else do we need to know: Minikube 0.31.0 worked, this happened right after upgrade.

[robrich]

Deleted ~/.minikube folder, rebooted, and it started. Not sure what it was, but I'll call it solved.

[ohalushchak]

I have run this command for version 0.33.1 and got a another error

It is completely new installation.

[tstromberg]

I ran into this today:

$ time ./out/minikube start --vm-driver=kvm2
😄  minikube v1.0.1 on linux (amd64)
🤹  Downloading Kubernetes v1.14.1 images in the background ...
💡  Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
🏃  Re-using the currently running kvm2 VM for "minikube" ...
⌛  Waiting for SSH access ...
📶  "minikube" IP address is 192.168.39.115
🐳  Configuring Docker as the container runtime ...
🐳  Version of container runtime is 18.06.3-ce
⌛  Waiting for image downloads to complete ...
✨  Preparing Kubernetes environment ...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x117f24a]

goroutine 688 [running]:
k8s.io/minikube/pkg/minikube/assets.(*BinDataAsset).Read(0xc000321000, 0xc0007ec000, 0x8000, 0x8000, 0x0, 0x7fcc88138100, 0x21)
	/usr/local/google/home/tstromberg/go/src/k8s.io/minikube/pkg/minikube/assets/vm_assets.go:230 +0x2a
io.copyBuffer(0x7fcc88138008, 0xc0000d2ba0, 0x7fcc88138118, 0xc000321000, 0xc0007ec000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/lib/google-golang/src/io/io.go:402 +0x122
io.Copy(...)
	/usr/lib/google-golang/src/io/io.go:364
k8s.io/minikube/pkg/minikube/bootstrapper.(*SSHRunner).Copy.func1(0xc000646450, 0x1798fe0, 0xc0000d2ba0, 0x17cb100, 0xc000321000)
	/usr/local/google/home/tstromberg/go/src/k8s.io/minikube/pkg/minikube/bootstrapper/ssh_runner.go:181 +0x35a
created by k8s.io/minikube/pkg/minikube/bootstrapper.(*SSHRunner).Copy
	/usr/local/google/home/tstromberg/go/src/k8s.io/minikube/pkg/minikube/bootstrapper/ssh_runner.go:176 +0x3d3
Command exited with non-zero status 2
7.48user 0.79system 0:12.17elapsed 67%CPU (0avgtext+0avgdata 38288maxresident)k
0inputs+72outputs (0major+767minor)pagefaults 0swaps

The strace output didn't yield any obvious problems to me, other than ssh complaining about something or another. Using -v=8 --alsologtostderr also didn't show too much:

I0501 11:46:17.451602   70519 ssh_runner.go:101] SSH: sudo mkdir -p /etc/kubernetes/addons
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x117f24a]

Nothing suspicious entries in dmesg either. I'll dig a little further.

[tstromberg]

The problem disappeared before I had a proper chance to debug it. As far as I can tell, it's just scp'ing data from an in memory []byte{} to scp, so I'm not exactly sure why it would crash. Searching around, other folks have hit a similar panic, and it's possible that it's related to negative or conflicting reads.

It feels like a race condition, but I was able to replicate it so many times that I'm not sure.

[tstromberg]

Increasing priority, since @medyagh also ran into this today.

[medyagh]

I tried deleting the ~/.minikube folder but it didn't make a difference for me.
here is my output :

 ./minikube-darwin-amd64 start --vm-driver hyperkit
😄  minikube v1.0.1 on darwin (amd64)
🤹  Downloading Kubernetes v1.14.1 images in the background ...
🔥  Creating hyperkit VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
📶  "minikube" IP address is 192.168.64.5
🐳  Configuring Docker as the container runtime ...
🐳  Version of container runtime is 18.06.3-ce
⌛  Waiting for image downloads to complete ...
✨  Preparing Kubernetes environment ...
💾  Downloading kubeadm v1.14.1
💾  Downloading kubelet v1.14.1
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x4c6c97a]

goroutine 903 [running]:
k8s.io/minikube/pkg/minikube/assets.(*BinDataAsset).Read(0xc00034ce00, 0xc000de6000, 0x8000, 0x8000, 0x0, 0x0, 0x1b)
	/Users/medya/go/src/k8s.io/minikube/pkg/minikube/assets/vm_assets.go:230 +0x2a
io.copyBuffer(0x6c64bb8, 0xc000d268c0, 0x6c64e08, 0xc00034ce00, 0xc000de6000, 0x8000, 0x8000, 0x0, 0xc000175380, 0x52ad0a0)
	/usr/local/go/src/io/io.go:402 +0x122
io.Copy(...)
	/usr/local/go/src/io/io.go:364
k8s.io/minikube/pkg/minikube/bootstrapper.(*SSHRunner).Copy.func1(0xc00060e7b0, 0x52b4a00, 0xc000d268c0, 0x52e73a0, 0xc00034ce00)
	/Users/medya/go/src/k8s.io/minikube/pkg/minikube/bootstrapper/ssh_runner.go:181 +0x35a
created by k8s.io/minikube/pkg/minikube/bootstrapper.(*SSHRunner).Copy
	/Users/medya/go/src/k8s.io/minikube/pkg/minikube/bootstrapper/ssh_runner.go:176 +0x3d3

and here is my minikube logs

 ./minikube-darwin-amd64 logs
==> dmesg <==
[May 3 22:28] ERROR: earlyprintk= earlyser already used
[  +0.000000] ACPI BIOS Warning (bug): Incorrect checksum in table [DSDT] - 0xC0, should be 0x1D (20170831/tbprint-211)
[  +0.000000] ACPI Error: Could not enable RealTimeClock event (20170831/evxfevnt-218)
[  +0.000003] ACPI Warning: Could not enable fixed event - RealTimeClock (4) (20170831/evxface-654)
[  +0.010195] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
[  +0.356575] systemd-fstab-generator[1038]: Ignoring "noauto" for root device
[  +0.002501] systemd[1]: File /usr/lib/systemd/system/systemd-journald.service:35 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
[  +0.000002] systemd[1]: Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
[  +0.585779] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
[  +0.889848] vboxguest: loading out-of-tree module taints kernel.
[  +0.003510] vboxguest: PCI device not found, probably running on physical hardware.
[  +3.423126] systemd-fstab-generator[1821]: Ignoring "noauto" for root device

==> kernel <==
 22:30:24 up 2 min,  0 users,  load average: 0.53, 0.29, 0.11
Linux minikube 4.15.0 #1 SMP Thu Apr 25 20:51:48 UTC 2019 x86_64 GNU/Linux

==> kubelet <==
-- Logs begin at Fri 2019-05-03 22:28:51 UTC, end at Fri 2019-05-03 22:30:24 UTC. --
-- No entries --

[afbjorklund]

Hmm, wonder if I missed a dependency when I removed the templating from the default target...