Recently built minikube fails to start with vmware driver

[markpeek]

What Happened?

Build both minikube and the amd64 ISO with recent changes and it fails to start with the vmware driver.

$ ./minikube-darwin-amd64 start --driver vmware --iso-url=file://$(pwd)/minikube-amd64.iso
😄  minikube v1.30.0 on Darwin 13.2.1
✨  Using the vmware driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
💾  Downloading Kubernetes v1.26.3 preload ...
    > preloaded-images-k8s-v18-v1...:  397.02 MiB / 397.02 MiB  100.00% 53.18 M
🔥  Creating vmware VM (CPUs=2, Memory=6000MB, Disk=20000MB) .../ ^C

From the logs this appears to be due to the docker user account being removed. Looking at commit history, it was removed via Disable plain text login for ISO by @medyagh.

Attach the log file

I0403 10:43:40.090076   20589 main.go:141] libmachine: (minikube) DBG | IP found in DHCP lease table: 192.168.111.164
I0403 10:43:40.090091   20589 main.go:141] libmachine: (minikube) DBG | Got an ip: 192.168.111.164
I0403 10:43:40.090737   20589 main.go:141] libmachine: (minikube) DBG | Creating Tar key bundle...
I0403 10:43:40.091232   20589 main.go:141] libmachine: (minikube) DBG | executing: /Users/mark/bin/vmrun -gu docker -gp tcuser directoryExistsInGuest /Users/mark/.minikube/machines/minikube/minikube.vmx /var/lib/boot2docker
- I0403 10:43:40.426605   20589 main.go:141] libmachine: (minikube) DBG | Error: Invalid user name or password for the guest OS
I0403 10:43:40.429937   20589 main.go:141] libmachine: (minikube) DBG | executing: /Users/mark/bin/vmrun -gu docker -gp tcuser CopyFileFromHostToGuest /Users/mark/.minikube/machines/minikube/minikube.vmx /Users/mark/.minikube/machines/minikube/userdata.tar /home/docker/userdata.tar
/ I0403 10:43:40.690751   20589 main.go:141] libmachine: (minikube) DBG | Error: Invalid user name or password for the guest OS
I0403 10:43:40.694008   20589 main.go:141] libmachine: (minikube) DBG | executing: /Users/mark/bin/vmrun -gu docker -gp tcuser runScriptInGuest /Users/mark/.minikube/machines/minikube/minikube.vmx /bin/sh sudo sh -c "tar xvf /home/docker/userdata.tar -C /home/docker > /var/log/userdata.log 2>&1 && chown -R docker:staff /home/docker"
\ I0403 10:43:40.954608   20589 main.go:141] libmachine: (minikube) DBG | Error: Invalid user name or password for the guest OS
I0403 10:43:40.957541   20589 main.go:141] libmachine: (minikube) DBG | executing: /Users/mark/bin/vmrun -gu docker -gp tcuser runScriptInGuest /Users/mark/.minikube/machines/minikube/minikube.vmx /bin/sh sudo /bin/mv /home/docker/userdata.tar /var/lib/boot2docker/userdata.tar
- I0403 10:43:41.219867   20589 main.go:141] libmachine: (minikube) DBG | Error: Invalid user name or password for the guest OS
I0403 10:43:41.223214   20589 main.go:141] libmachine: (minikube) DBG | executing: /Users/mark/bin/vmrun -gu docker -gp tcuser enableSharedFolders /Users/mark/.minikube/machines/minikube/minikube.vmx
/ I0403 10:43:41.497243   20589 main.go:141] libmachine: (minikube) DBG | executing: /Users/mark/bin/vmrun -gu docker -gp tcuser addSharedFolder /Users/mark/.minikube/machines/minikube/minikube.vmx Users /Users
| I0403 10:43:41.777151   20589 main.go:141] libmachine: (minikube) DBG | executing: /Users/mark/bin/vmrun -gu docker -gp tcuser runScriptInGuest /Users/mark/.minikube/machines/minikube/minikube.vmx /bin/sh [ ! -d /hosthome ]&& sudo mkdir /hosthome; sudo mount --bind /mnt/hgfs//hosthome /hosthome || [ -f /usr/local/bin/vmhgfs-fuse ]&& sudo /usr/local/bin/vmhgfs-fuse -o allow_other .host:/Users /hosthome || sudo mount -t vmhgfs -o uid=$(id -u),gid=$(id -g) .host:/Users /hosthome
- I0403 10:43:42.041025   20589 main.go:141] libmachine: (minikube) DBG | Error: Invalid user name or password for the guest OS
- ```

### Operating System

macOS (Default)

### Driver

VMware

[afbjorklund]

Sorry about that, it needs to use the ssh key as provided in the image.

Error: Invalid user name or password for the guest OS

It seems to be created correctly (in the tar), in the vmware driver:

https://github.com/machine-drivers/docker-machine-driver-vmware/blob/v0.1.5/pkg/drivers/vmware/driver.go#L624

[afbjorklund]

The automounter will pick the information up, from the disk:

https://github.com/kubernetes/minikube/blob/master/deploy/iso/minikube-iso/package/automount/minikube-automount

After that, the ssh key is supposed to be installed and work.

/home/docker/.ssh/authorized_keys

[markpeek]

Thank you for the pointers and confirming the regression. Noted about needing to rewrite the vmware driver to support injecting the SSH key via cloud-init.

The use of the docker login appears to be inherited from the older Docker machine vmwarefusion driver to inject the SSH keys. As the docker login was documented in the original boot2docker project, are there any other machine drivers that rely on this same behavior that might be broken with the minikube 1.30 release?

[afbjorklund]

are there any other machine drivers that rely on this same behavior that might be broken with the minikube 1.30 release?

it is possible, but I couldn't find any use of "password" in the other drivers from a casual lookthrough right now (grep).

	// Register all of the drvs we know of
	_ "k8s.io/minikube/pkg/minikube/registry/drvs/docker"
	_ "k8s.io/minikube/pkg/minikube/registry/drvs/hyperkit"
	_ "k8s.io/minikube/pkg/minikube/registry/drvs/hyperv"
	_ "k8s.io/minikube/pkg/minikube/registry/drvs/kvm2"
	_ "k8s.io/minikube/pkg/minikube/registry/drvs/none"
	_ "k8s.io/minikube/pkg/minikube/registry/drvs/parallels"
	_ "k8s.io/minikube/pkg/minikube/registry/drvs/podman"
	_ "k8s.io/minikube/pkg/minikube/registry/drvs/qemu2"
	_ "k8s.io/minikube/pkg/minikube/registry/drvs/ssh"
	_ "k8s.io/minikube/pkg/minikube/registry/drvs/virtualbox"
	_ "k8s.io/minikube/pkg/minikube/registry/drvs/vmware"
	_ "k8s.io/minikube/pkg/minikube/registry/drvs/vmwarefusion"

You are right that it was inherited from boot2docker, which in turn inherited the login from tinycore (thus the T.C.)

[afbjorklund]

Noted about needing to rewrite the vmware driver to support injecting the SSH key via cloud-init.

Either using cloud-init and the user-data, or through the old docker-machine approach with tar ?

There is no support for cloud-init in minikube, so I guess that would be mostly for docker-machine ?

The ubuntu cloud images should come with support, for adding the user and the key with config.

[medyagh]

VMWare is not our officially supported/tested driver but it seems like this can be fixed by adding it to the machine driver lib.
I would accept and review any PR that would fix this issue.

till then the only solution for vmware drivers would be downgrading to an older minikube version.

[marshalc]

You might want to update the docs to reflect this unsupported status - https://minikube.sigs.k8s.io/docs/drivers/vmware/ . I'd just gone through the guide to use this, only to find at the cli that the unsupported message came up.

[lbogdan]

I'm interested in fixing this.

@afbjorklund @medyagh Please correct me if I'm wrong, but from what I understand, with only looking at the error logs in the issue description and reading the comments, the issue is with provisioning /home/docker/.ssh/authorized_keys inside a freshly created VM, where the VMware driver would use vmrun -gu docker -gp tcuser ... to run operations inside the VM, but in the latest boot2docker minikube ISO image, the password for the docker user was disabled, so running vmrun to provision authorized_keys fails.

That being said, can you please explain the approach / point me to the code of a working driver where the authorized_keys provisioning is done? Thanks!

[lbogdan]

I was also thinking, instead of disabling the password in the ISO, can't we keep it, and disable it first thing, as part of VM initialization?

[afbjorklund]

That being said, can you please explain the approach / point me to the code of a working driver where the authorized_keys provisioning is done? Thanks!

It is inserted into the disk image, like so: https://github.com/kubernetes/minikube/blob/v1.30.1/pkg/drivers/qemu/qemu.go#L643

Then it is extracted, during the automount:
https://github.com/kubernetes/minikube/blob/v1.30.1/deploy/iso/minikube-iso/package/automount/minikube-automount#L32

[afbjorklund]

@lbogdan: I made PR to move the external driver in

• Use a built-in version of the vmware driver #16687

That should make it faster to iterate, since it doesn't need an external docker-machine-driver-vmware

The last release from https://github.com/machine-drivers/docker-machine-driver-vmware was from 2021

[afbjorklund]

I think it should be possible to format and mount the vmdk, with something like FAT32 similar to a cidata.iso

Then you could extract the userdata.tar from this temporary partition, before reformatting the whole "disk"?

[lbogdan]

So basically the automounter expects the first 4096 bytes of the raw partition to be "boot2docker, please format-me" followed by a tar stream containing .ssh/authorized_keys[2] (any reason for the authorized_keys2?)?

Unfortunately, I don't think we can doctor a .vmdk image this way with just off the shelf Windows / macOS tooling.

[afbjorklund]

Unfortunately, I don't think we can doctor a .vmdk image this way with just off the shelf Windows / macOS tooling.

No, it would need a separate code path. Then again, it seemed like it already had one ? (in the automounter, that is)

[afbjorklund]

any reason for the authorized_keys2?

Ancient history, I think. Deprecated since 3.0 (2001)

[lbogdan]

Then again, it seemed like it already had one ? (in the automounter, that is)

Yeah, but that's assuming the authorized_keys provisioning is handled separately, which is done in the VMware driver here: https://github.com/machine-drivers/docker-machine-driver-vmware/blob/b30963f81e6e25743b42d469a22a5a9f5db742fb/pkg/drivers/vmware/driver.go#L262-L332 .

Note that is supports a cloud-init config drive (if d.ConfigDriveURL != "" {), but even so, it still SSHes into the VM using a password to copy over authorized_keys: https://github.com/machine-drivers/docker-machine-driver-vmware/blob/b30963f81e6e25743b42d469a22a5a9f5db742fb/pkg/drivers/vmware/driver.go#L650-L659 .

What about my other suggestion?

I was also thinking, instead of disabling the password in the ISO, can't we keep it, and disable it first thing, as part of VM initialization?

[lbogdan]

Hm, I might have found a way to create a .vmdk disk with the automounter expected format:

• first, create a 1MB (that's the minimum) preallocated disk:

  vmware-vdiskmanager -c -s 1MB -t 2 test.vmdk

  this creates two files, test.vmdk, containing disk metadata, and test-flat.vmdk, containing 1_048_576 (=1MB) 0 bytes, that is the raw disk data

• write the tar stream to the beginning of test-flat.vmdk

• convert the disk to a growable disk:

   vmware-vdiskmanager -r test.vmdk -t 0 test2.vmdk

  this creates a single test2.vmdk file, containing both metadata and raw data

• grow the converted disk to the desired size:

  vmware-vdiskmanager -x 10GB test2.vmdk

• delete test[-flat].vmdk

• use test2.vmdk as the VM disk.

It's a bit convoluted, but it should work.

[lbogdan]

Did the whole process manually, attached the disk to an existing VM, and lo and behold, it works!

# lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
[...]
sdb       8:16   0   10G  0 disk

# dd if=/dev/sdb bs=1 count=29
boot2docker, please format-me29+0 records in
29+0 records out
29 bytes copied, 0.000151799 s, 191 kB/s

[lbogdan]

@afbjorklund So where should this be fixed? Even with #16687 , it looks like the changes would still have to go to upstream https://github.com/machine-drivers/docker-machine-driver-vmware ?

[afbjorklund]

Well, it would have to go to a fork of that repo. If you want to go all-in (for developing), you could copy stuff:

e4ccde5

We want to drop the dependency on docker-machine, so that we are free to evolve the "libmachine" API
So there is a work-in-progress, to move to a separate entity called "minikube-machine" to offer libmachine

The code/binary in docker-machine-driver-vmware, is supposed to work with both docker-machine and minikube
That would of course also be a way of testing it, to run with docker-machine create -d vmware and the binary

[ErlinDez]

@lbogdan how do you go about writing the tar stream to the beginning of test-flat.vmdk
?

[lbogdan]

I did that manually, but I guess the same as overwriting (part of) a file? Open, seek to the beginning, write bytes, close.

[lbogdan]

Was there any progress on this? If not, I'm planning on working on it this weekend.