hyperkit clock falls behind: failed to write or validate certificate: the certificate is not valid yet

[vincent-dm]

Minikube version: v0.18.0

Environment:

• OS: MacOS Sierra (10.12.4 (16E195))
• VM Driver : xhyve
• ISO version : v0.18.0

What happened:
When I wake my laptop after sleep, the clock of the Minikube VM lags behind. This causes problems with the registry credentials plugin, since AWS rejects credentials requests with an invalid timestamp.

What you expected to happen:
The clock should sync after computer wakeup.

How to reproduce it (as minimally and precisely as possible):
See above.

[aaron-prindle]

Thanks for reporting this. This seems to be a know issue for users w/ xhyve + docker:

docker/for-mac#17
https://forums.docker.com/t/time-in-container-is-out-of-sync/16566/8
For now you should be able to minikube ssh into the VM and then run:
docker run -it --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y)
to sync the time back after you sleep.

[aaron-prindle]

It seems that you can automate this by using sleepwatcher:

brew install sleepwatcher
brew services start sleepwatcher
echo minikube ssh -- docker run -it --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y) > ~/.wakeup
chmod +x ~/.wakeup

[r2d4]

ref moby/hyperkit#110

[vincent-dm]

@aaron-prindle thanks for the workaround, but unfortunately I can't get it working.
When I minikube ssh into the VM and run docker run -it --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y) it pulls an alpine image and then nsenter throws an error:

$ minikube ssh
$ docker run -it --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y)
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine

627beaf3eaaf: Already exists
Digest: sha256:58e1a1bb75db1b5a24a462dd5e2915277ea06438c3f105138f97eb53149673c4
Status: Downloaded newer image for alpine:latest
BusyBox v1.25.1 (2016-10-26 16:15:20 GMT) multi-call binary.

Usage: nsenter [OPTIONS] [PROG [ARGS]]

	-t, --target=PID		Target process to get namespaces from
	-m, --mount[=FILE]		Enter mount namespace
	-u, --uts[=FILE]		Enter UTS namespace (hostname etc)
	-i, --ipc[=FILE]		Enter System V IPC namespace
	-n, --net[=FILE]		Enter network namespace
	-p, --pid[=FILE]		Enter pid namespace
	-U, --user[=FILE]		Enter user namespace
	-S, --setuid=UID		Set uid in entered namespace
	-G, --setgid=GID		Set gid in entered namespace
	--preserve-credentials		Don't touch uids or gids
	-r, --root[=DIR]		Set root directory
	-w, --wd[=DIR]			Set working directory
	-F, --no-fork			Don't fork before exec'ing PROG

I'm also a bit in the dark about how this is supposed to work: as I understand it, the Minikube VM has a skewed clock, so how is launching a pod going to solve this?

Wouldn't it be more logical to run some kind of command directly in the Minikube VM to fix its clock? (like fetching the time from a time server)

I tried to come up with a working command myself, but all the usual linux commands seem to be missing from the flavour of Linux that the Minikube VM is running...

[aaron-prindle]

I actually tested this command w/ debian and incorrectly assumed it would be the same on alpine. Also I realize now that this must be a single command as the host terminal needs to complete the date to update in the vm. The following command should work properly:
minikube ssh -- docker run -i --rm --privileged --pid=host debian nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y)

This is just a workaround for now. We are considering making time-syncing daemon for the VM to solve this but this is not really directly related to minikube.

[vincent-dm]

Thanks, this workaround does the trick!

[sebgoa]

Ran into this on a VirtualBox minikube 0.19.0 as well.

The command did the trick

[tristanpemble]

I'm playing with using a privileged pod running ntpd to keep my Minikube VM's clock in sync. If it works well, I will try to submit a PR to bundle it as an addon (if that aligns with how you envisioned a fix)

[gavrie]

If you use Homebrew to install minikube and kubectl, you may have to add /usr/local/bin to your path in the ~/.wakeup script:

PATH=$PATH:/usr/local/bin
minikube ssh -- docker run -i --rm --privileged --pid=host debian nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y)

I also added the following line to set the timezone of the VM to the one of my Mac:

minikube ssh -- sudo ln -sf /usr/share/zoneinfo/Asia/Jerusalem /etc/localtime

(Instead of Asia/Jerusalem, you may want to determine your own timezone by running sudo systemsetup -gettimezone).

[bboreham]

out of interest, why run docker run --privileged ... date instead of sudo date ?

[hasan-abdul-rahman]

@aaron-prindle
Please note, your echo command will hardcode a specific date value into the ~/.wakeup script.

$ echo minikube ssh -- docker run -it --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y)
minikube ssh -- docker run -it --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u 102505152017

The whole thing should be surrounded in single quotes to prevent bash from expanding the date command.

$ echo 'minikube ssh -- docker run -it --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y)'
minikube ssh -- docker run -it --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y)

[vincent-dm]

After combining all comments above, I have this setup script:

brew install sleepwatcher
brew services start sleepwatcher
echo "PATH=\$PATH:/usr/local/bin
minikube ssh -- docker run -i --rm --privileged --pid=host debian nsenter -t 1 -m -u -n -i hwclock -s" > ~/.wakeup
chmod +x ~/.wakeup

[coderanger]

Since the minikube distro is already using systemd, might be nice to include timesyncd too. Wouldn't cover fixing drift when not on the internet, but would help for a lot of people I bet. I tried enabling it manually but it looks like that daemon isn't included in the minikube distro currently.

[eecue]

+1 this one is a pain when doing AWS dev as our certs get out of sync and fail: Signature expired: 20171114T003822Z is now earlier than 20171114T004302Z (20171114T004802Z - 5 min.)

[luigi-riefolo]

any updates on this issue?

[Morriz]

please?

I am just running virtualbox, but also have time out of sync...why is this thread just about xhyve when there seems no timesync daemon on the distro?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[mildebrandt]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[mildebrandt]

/remove-lifecycle stale

[msabramo]

I would add the seconds to the date command, so that it's more precise:

minikube ssh -- docker run -it --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y.%S)

but then I wonder what the purpose of docker run of nsenter is? It seems like one could use sudo date -- e.g.:

$ minikube ssh -- date
Wed Jul  4 11:23:15 UTC 2018

$ minikube ssh -- sudo date -u $(date -u +%m%d%H%M%Y.%S)
Tue Jul 10 15:27:28 UTC 2018

$ minikube ssh -- date
Tue Jul 10 15:27:39 UTC 2018

[tstromberg]

I'm not sure when it was added, but the minikube guest VM should include time synchronization. Apparently it works for me in kvm2:

$ timedatectl 
                      Local time: Wed 2018-09-19 22:24:46 UTC
                  Universal time: Wed 2018-09-19 22:24:46 UTC
                        RTC time: Wed 2018-09-19 22:24:46
                       Time zone: Etc/UTC (UTC, +0000)
       System clock synchronized: yes
systemd-timesyncd.service active: yes
                 RTC in local TZ: no

YMMV based on your environment.

[reegnz]

@tstromberg I see that minikube had systemd-timesyncd enabled but then you disabled it in #3476.
Could you elaborate on why you chose to disable it? Having an sntp daemon constantly running would definitely help in resolving this issue.

[afbjorklund]

One could argue that it is the job of the hypervisor to keep the time synced. Then again, some of them fail...

[tstromberg]

@reegnz - I disabled systemd-timesyncd because it could conflict with hypervisor time synchronization. At the time, I wasn't aware that there were hypervisors which didn't handle time synchronization out-of-band.

I agree though that for hyperkit, we will need an sntp daemon to talk to in order to make this work.

[reegnz]

@afbjorklund one could also argue that a vm should be blind to the hypervisor and take time from another trusted source instead.

The point is, we cannot fix the hypervisor as easily as we can fix this in the minikube ISO however. Once somebody fixes this in hyperkit we could get away with not having an sntpd running.

@tstromberg could we detect what hypervisor we are running in?

[reegnz]

Just checked systemd unit config parameters:
https://www.freedesktop.org/software/systemd/man/systemd.unit.html

One could use ConditionVirtualization to only activate for certain hypervisor types.

[reegnz]

Checked out the systemd-detect-virt code, hyperkit should be identifyable as bhyve (xhyve under hyperkit is a bhyve fork).
https://github.com/systemd/systemd/blob/64ef83139cf84a700b95e8150a458b0bb9f720de/src/basic/virt.c#L132

@tstromberg I think we should enable systemd-timesyncd and add a drop-in unit config with ConditionVirtualization=bhyve.

What do you think, should I give it a go?

[yhekma]

FYI I have encountered the same issue using vmware fusion.

[reegnz]

We could define multiple virtualization technologies in the ConditionVirtualization option, eg. ConditionVirtualization=bhyve|vmware.

I am looking into how I can enable timesyncd with buildroot and build the image on mac (don't have a linux machine at hand).

I will come back with the results.

Then somebody could help out how we get the drop-in file into the buildroot build (maybe with a package 'systemd-timesyncd-conditional'?).

[danielsonjh]

I've encountered the same issue (clock drift when after sleeping) on Windows 10 with the hyper-v driver as well

[reegnz]

@danielsonjh thanks for the input! So we're at 3 hypervisors now...

@tstromberg would it make sense to just reenable the systemd-timesyncd daemon and open a follow-up ticket for the virtualbox issue (if it even exists anymore)?

3 out of 4 hypervisors break because of no ntp in the vm anymore.

We can then experiment with the ConditionVirtualization=!oracle unit file config.

[medyagh]

I think due to the multiple related issues we have seen, we need to address this as an important bug to fix soon.

[cpoole]

I have also encountered this running with kvm2. So that makes 4 hypervisors

minikube v1.2.0

[rkleine]

Just the some workaround but in one line:

ssh -i ~/.minikube/machines/minikube/id_rsa docker@$(minikube ip) "docker run --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y)"

[asychev]

Unfortunately, does not work for me. After MacBook wake up today:

$ date
Tue Sep 24 09:35:08 CEST 2019

$ minikube ssh date
Mon Sep 23 19:16:38 UTC 2019

Minikube 1.4.0, hyperkit driver

[gWOLF3]

Yep, I just ran into this issue.

I'm not sure this issue should be closed.

[BatmanAoD]

@reegnz My coworkers and I are also observing this behavior on VirtualBox. Looking through this thread, I don't see why VirtualBox is excluded from the systemd-timesyncd fix; I see that @dcecile said they hadn't observed drift with VirtualBox, but other commenters did.

[phoewass]

I use Minikube on Windows 10 with VirtualBox. Everytime I lock the screen the time in minikube gets out of sync with the host machine.
I tried executing the commands provided by @aaron-prindle and it didn't sync the date.

[rajasaur]

yes, confirmed with VirtualBox and minikube, the drift gets bigger depending on how long the machine is in sleep mode. @msabramo 's comment minikube ssh -- docker run -it --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i date -u $(date -u +%m%d%H%M%Y.%S) works though

[Diggsey]

I use Minikube on Windows 10 with VirtualBox and I see the same issue. The commands provided by @aaron-prindle DO NOT sync the date.