Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update kvm docker machine base image #15628

Merged

Conversation

spowelljr
Copy link
Member

@spowelljr spowelljr commented Jan 11, 2023

  • Updated without_lxc to libvirt_without_lxc as per https://gitlab.com/libvirt/libvirt-go-module/-/merge_requests/34
  • Updated base image of docker-machine-driver-kvm2-amd64 builder from Ubuntu 16.04 to Ubuntu 20.04 to fix libvirt build error
  • Added docker system prune -a --volumes -f to cross build to clear out previous run artifacts

This makes the build on #15537 complete

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jan 11, 2023
@spowelljr
Copy link
Member Author

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jan 11, 2023
@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 11, 2023
@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15628) |
+----------------+----------+---------------------+
| minikube start | 52.8s    | 52.5s               |
| enable ingress | 26.1s    | 28.3s               |
+----------------+----------+---------------------+

Times for minikube start: 51.5s 51.7s 53.8s 53.0s 54.3s
Times for minikube (PR 15628) start: 51.6s 52.7s 53.9s 51.8s 52.8s

Times for minikube ingress: 27.2s 25.6s 23.7s 26.7s 27.2s
Times for minikube (PR 15628) ingress: 28.2s 27.7s 29.2s 27.7s 28.7s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15628) |
+----------------+----------+---------------------+
| minikube start | 25.6s    | 25.0s               |
| enable ingress | 20.7s    | 20.4s               |
+----------------+----------+---------------------+

Times for minikube start: 26.3s 25.7s 24.4s 25.7s 26.0s
Times for minikube (PR 15628) start: 25.0s 26.2s 25.6s 23.4s 24.9s

Times for minikube ingress: 21.9s 19.9s 20.5s 22.4s 18.9s
Times for minikube (PR 15628) ingress: 19.5s 20.4s 21.4s 19.5s 20.9s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15628) |
+----------------+----------+---------------------+
| minikube start | 21.9s    | 22.3s               |
| enable ingress | 26.2s    | 26.4s               |
+----------------+----------+---------------------+

Times for minikube start: 22.2s 22.3s 22.0s 21.5s 21.6s
Times for minikube (PR 15628) start: 21.1s 22.1s 24.9s 22.1s 21.4s

Times for minikube ingress: 26.4s 26.0s 26.4s 25.9s 26.4s
Times for minikube (PR 15628) ingress: 26.4s 26.4s 26.4s 26.4s 26.4s

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Docker_macOS TestPause/serial/Pause (gopogh) 0.00 (chart)
Docker_Windows TestKicStaticIP (gopogh) 0.00 (chart)
KVM_Linux_containerd TestFunctional/serial/LogsFileCmd (gopogh) 0.61 (chart)
KVM_Linux TestStartStop/group/default-k8s-diff-port/serial/AddonExistsAfterStop (gopogh) 0.88 (chart)
KVM_Linux TestStartStop/group/default-k8s-diff-port/serial/Pause (gopogh) 0.88 (chart)
KVM_Linux TestStartStop/group/default-k8s-diff-port/serial/SecondStart (gopogh) 0.88 (chart)
KVM_Linux TestStartStop/group/default-k8s-diff-port/serial/UserAppExistsAfterStop (gopogh) 0.88 (chart)
KVM_Linux TestStartStop/group/default-k8s-diff-port/serial/VerifyKubernetesImages (gopogh) 0.88 (chart)
Docker_Linux TestNetworkPlugins/group/kindnet/Start (gopogh) 5.98 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/DeployApp (gopogh) 11.57 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/FirstStart (gopogh) 11.57 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/UserAppExistsAfterStop (gopogh) 11.57 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/SecondStart (gopogh) 13.01 (chart)
Docker_Windows TestPause/serial/SecondStartNoReconfiguration (gopogh) 23.08 (chart)
Docker_macOS TestMultiNode/serial/RestartMultiNode (gopogh) 23.23 (chart)
Docker_macOS TestStartStop/group/newest-cni/serial/Pause (gopogh) 27.27 (chart)
Docker_Windows TestNetworkPlugins/group/false/DNS (gopogh) 30.77 (chart)
Docker_Windows TestNetworkPlugins/group/kubenet/DNS (gopogh) 30.77 (chart)
Docker_Windows TestStartStop/group/newest-cni/serial/Pause (gopogh) 30.77 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/DeployApp (gopogh) 42.15 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/UserAppExistsAfterStop (gopogh) 42.15 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/FirstStart (gopogh) 42.62 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/SecondStart (gopogh) 43.09 (chart)
Docker_Windows TestNetworkPlugins/group/bridge/DNS (gopogh) 46.15 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-diff-port/serial/UserAppExistsAfterStop (gopogh) 47.93 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-diff-port/serial/SecondStart (gopogh) 48.78 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-diff-port/serial/DeployApp (gopogh) 49.19 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-diff-port/serial/FirstStart (gopogh) 50.00 (chart)
Hyper-V_Windows TestNetworkPlugins/group/kubenet/Start (gopogh) 50.00 (chart)
Docker_macOS TestMultiNode/serial/RestartKeepsNodes (gopogh) 52.26 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@spowelljr
Copy link
Member Author

There seems there might be something weird going on in CI.

If I build the docker-machine-driver-kvm2-amd64 binary locally (but through docker) the libvirt version is 6.0.0, which is expected.

But if I download https://storage.googleapis.com/minikube-builds/15628/docker-machine-driver-kvm2-amd64 the libvirt version is still old (1.2.19). However the LXC versions are correctly removed, so the build is successful but it's like it's using an old version of the image (Ubuntu 16) to build the with.

Investigating what might be going on

@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jan 11, 2023
@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15628) |
+----------------+----------+---------------------+
| minikube start | 54.3s    | 55.6s               |
| enable ingress | 27.8s    | 26.5s               |
+----------------+----------+---------------------+

Times for minikube start: 54.4s 53.2s 55.0s 53.8s 54.9s
Times for minikube (PR 15628) start: 53.9s 54.9s 58.6s 55.1s 55.6s

Times for minikube (PR 15628) ingress: 25.2s 28.7s 28.7s 24.7s 25.2s
Times for minikube ingress: 28.2s 25.6s 28.7s 28.2s 28.2s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15628) |
+----------------+----------+---------------------+
| minikube start | 26.5s    | 26.2s               |
| enable ingress | 21.3s    | 20.2s               |
+----------------+----------+---------------------+

Times for minikube start: 25.6s 25.1s 27.8s 27.1s 27.1s
Times for minikube (PR 15628) start: 26.0s 26.6s 25.1s 28.4s 24.9s

Times for minikube ingress: 23.0s 20.0s 21.0s 21.0s 21.5s
Times for minikube (PR 15628) ingress: 21.0s 19.5s 21.0s 20.0s 19.5s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15628) |
+----------------+----------+---------------------+
| minikube start | 25.1s    | 25.8s               |
| enable ingress | 26.5s    | 26.3s               |
+----------------+----------+---------------------+

Times for minikube start: 22.1s 21.5s 22.6s 33.3s 26.2s
Times for minikube (PR 15628) start: 23.0s 32.9s 25.0s 23.0s 25.2s

Times for minikube ingress: 26.5s 26.5s 26.5s 26.5s 26.5s
Times for minikube (PR 15628) ingress: 26.0s 26.5s 26.0s 26.5s 26.5s

@spowelljr
Copy link
Member Author

With debugging I was able to confirm that the file contains the expected libvirt version in CI, but the uploaded file has a different libvirt version, will continue investigating

@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15628) |
+----------------+----------+---------------------+
| minikube start | 54.9s    | 54.9s               |
| enable ingress | 27.5s    | 25.9s               |
+----------------+----------+---------------------+

Times for minikube start: 53.4s 56.4s 56.4s 54.0s 54.4s
Times for minikube (PR 15628) start: 55.0s 56.5s 53.2s 54.1s 55.8s

Times for minikube ingress: 25.2s 26.7s 27.1s 23.1s 35.2s
Times for minikube (PR 15628) ingress: 27.7s 28.6s 24.2s 25.2s 23.7s

docker driver with docker runtime
error collecting results for docker driver: timing run 0 with minikube: timing cmd: [out/minikube addons enable ingress]: waiting for minikube: exit status 10
docker driver with containerd runtime
error downloading artifacts: artifact download start: exit status 90

@spowelljr
Copy link
Member Author

spowelljr commented Jan 11, 2023

Confirmed that the binary shasum in CI doesn't match the file uploaded
276b2b9060f961071b5531c81749064ca8783fc1b577e1e2b8db9a323d51daff out/docker-machine-driver-kvm2-amd64
be3d176c1eb1ad0ea8b738466339d2469679119667bbd2925cad4efd80e06a6d ./docker-machine-driver-kvm2-amd64

@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15628) |
+----------------+----------+---------------------+
| minikube start | 55.8s    | 55.2s               |
| enable ingress | 29.2s    | 27.9s               |
+----------------+----------+---------------------+

Times for minikube start: 56.6s 53.8s 56.0s 58.0s 54.7s
Times for minikube (PR 15628) start: 54.6s 55.6s 55.4s 55.6s 55.0s

Times for minikube ingress: 28.8s 28.8s 29.8s 29.7s 28.8s
Times for minikube (PR 15628) ingress: 25.7s 29.2s 28.7s 27.7s 28.3s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15628) |
+----------------+----------+---------------------+
| minikube start | 27.0s    | 26.3s               |
| enable ingress | 26.7s    | 20.7s               |
+----------------+----------+---------------------+

Times for minikube ingress: 21.0s 20.5s 21.5s 20.5s 50.0s
Times for minikube (PR 15628) ingress: 22.0s 19.5s 20.5s 20.5s 21.0s

Times for minikube (PR 15628) start: 26.2s 27.4s 26.1s 25.8s 26.2s
Times for minikube start: 28.6s 26.8s 26.8s 27.0s 25.9s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15628) |
+----------------+----------+---------------------+
| minikube start | 22.8s    | 24.0s               |
| enable ingress | 26.1s    | 26.0s               |
+----------------+----------+---------------------+

Times for minikube start: 22.1s 22.7s 23.7s 22.7s 22.6s
Times for minikube (PR 15628) start: 25.6s 23.6s 23.4s 23.3s 24.1s

Times for minikube (PR 15628) ingress: 26.0s 26.0s 26.0s 26.0s 26.0s
Times for minikube ingress: 26.5s 26.0s 26.0s 26.0s 26.0s

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Hyper-V_Windows TestNetworkPlugins/group/bridge/NetCatPod (gopogh) 0.00 (chart)
Docker_Windows TestMultiNode/serial/RestartKeepsNodes (gopogh) 5.88 (chart)
Docker_Linux_containerd TestFunctional/serial/LogsFileCmd (gopogh) 7.14 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/DeployApp (gopogh) 11.29 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/FirstStart (gopogh) 11.29 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/UserAppExistsAfterStop (gopogh) 11.29 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/SecondStart (gopogh) 12.70 (chart)
Docker_macOS TestMultiNode/serial/RestartMultiNode (gopogh) 24.53 (chart)
Docker_Linux_containerd TestMultiNode/serial/RestartKeepsNodes (gopogh) 27.38 (chart)
Docker_Windows TestNetworkPlugins/group/enable-default-cni/DNS (gopogh) 31.25 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/UserAppExistsAfterStop (gopogh) 41.94 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/FirstStart (gopogh) 42.40 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/DeployApp (gopogh) 42.86 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/SecondStart (gopogh) 43.31 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-diff-port/serial/UserAppExistsAfterStop (gopogh) 48.39 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-diff-port/serial/SecondStart (gopogh) 49.21 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-diff-port/serial/DeployApp (gopogh) 49.61 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-diff-port/serial/FirstStart (gopogh) 50.39 (chart)
Docker_macOS TestMultiNode/serial/RestartKeepsNodes (gopogh) 51.57 (chart)
Docker_Windows TestNetworkPlugins/group/bridge/DNS (gopogh) 52.94 (chart)
Docker_Linux TestNetworkPlugins/group/calico/Start (gopogh) 73.11 (chart)
Docker_Linux_containerd TestNetworkPlugins/group/bridge/DNS (gopogh) 77.44 (chart)
Docker_Windows TestNetworkPlugins/group/kubenet/HairPin (gopogh) 83.33 (chart)
Docker_Linux TestNetworkPlugins/group/false/DNS (gopogh) 84.03 (chart)
Docker_Linux TestNetworkPlugins/group/bridge/DNS (gopogh) 84.87 (chart)
Docker_Linux TestNetworkPlugins/group/enable-default-cni/DNS (gopogh) 85.71 (chart)
Docker_Linux TestNetworkPlugins/group/kubenet/DNS (gopogh) 88.24 (chart)
Docker_Windows TestNetworkPlugins/group/calico/Start (gopogh) 88.24 (chart)
Docker_Linux_containerd TestNetworkPlugins/group/calico/Start (gopogh) 98.40 (chart)
Docker_macOS TestIngressAddonLegacy/serial/ValidateIngressDNSAddonActivation (gopogh) 98.74 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@spowelljr
Copy link
Member Author

spowelljr commented Jan 11, 2023

Discovered the root of the problem, we aren't purging Docker on build in Jenkins so when it checks for the existing image it exists and doesn't rebuild it, resulting in my Dockerfile changes not being built and used. Will remove all the debugging changes and update CI to purge Docker on start.

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh, spowelljr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15628) |
+----------------+----------+---------------------+
| minikube start | 55.3s    | 55.4s               |
| enable ingress | 26.1s    | 26.7s               |
+----------------+----------+---------------------+

Times for minikube ingress: 27.8s 25.1s 25.2s 23.2s 29.1s
Times for minikube (PR 15628) ingress: 28.2s 26.2s 24.7s 27.6s 26.8s

Times for minikube start: 55.4s 54.3s 55.2s 55.0s 56.4s
Times for minikube (PR 15628) start: 55.3s 55.7s 57.9s 53.7s 54.4s

docker driver with docker runtime

+-------------------+----------+---------------------+
|      COMMAND      | MINIKUBE | MINIKUBE (PR 15628) |
+-------------------+----------+---------------------+
| minikube start    | 26.0s    | 26.4s               |
| ⚠️  enable ingress | 22.1s    | 32.7s ⚠️             |
+-------------------+----------+---------------------+

Times for minikube start: 26.9s 26.6s 25.3s 25.7s 25.4s
Times for minikube (PR 15628) start: 28.2s 24.9s 25.4s 25.5s 27.7s

Times for minikube ingress: 20.9s 25.4s 20.5s 21.5s 22.0s
Times for minikube (PR 15628) ingress: 20.5s 19.0s 19.9s 23.0s 81.0s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15628) |
+----------------+----------+---------------------+
| minikube start | 24.2s    | 23.5s               |
| enable ingress | 26.3s    | 26.2s               |
+----------------+----------+---------------------+

Times for minikube start: 22.5s 21.4s 22.1s 22.0s 33.0s
Times for minikube (PR 15628) start: 22.0s 22.4s 25.0s 22.7s 25.5s

Times for minikube ingress: 26.5s 26.4s 26.4s 26.4s 25.9s
Times for minikube (PR 15628) ingress: 26.4s 26.5s 25.9s 26.5s 26.0s

@spowelljr spowelljr merged commit 7c75349 into kubernetes:master Jan 12, 2023
@spowelljr spowelljr deleted the updateKVMDockerMachineBaseImage branch January 12, 2023 03:23
@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Hyper-V_Windows TestErrorSpam/setup (gopogh) 0.00 (chart)
Hyper-V_Windows TestMultiNode/serial/DeployApp2Nodes (gopogh) 0.00 (chart)
Docker_Linux_containerd TestNetworkPlugins/group/kindnet/Start (gopogh) 5.79 (chart)
Docker_Windows TestStartStop/group/newest-cni/serial/Pause (gopogh) 25.00 (chart)
Docker_Windows TestNetworkPlugins/group/enable-default-cni/DNS (gopogh) 31.58 (chart)
Hyper-V_Windows TestNoKubernetes/serial/StartWithK8s (gopogh) 33.33 (chart)
Hyper-V_Windows TestPause/serial/SecondStartNoReconfiguration (gopogh) 33.33 (chart)
Docker_Windows TestNetworkPlugins/group/kubenet/DNS (gopogh) 35.00 (chart)
KVM_Linux TestMultiNode/serial/RestartMultiNode (gopogh) 37.27 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/UserAppExistsAfterStop (gopogh) 42.98 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/FirstStart (gopogh) 43.44 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/DeployApp (gopogh) 43.90 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/SecondStart (gopogh) 43.90 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-diff-port/serial/UserAppExistsAfterStop (gopogh) 47.93 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-diff-port/serial/SecondStart (gopogh) 48.78 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-diff-port/serial/DeployApp (gopogh) 49.60 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-diff-port/serial/FirstStart (gopogh) 50.00 (chart)
Docker_Linux_containerd TestNetworkPlugins/group/enable-default-cni/DNS (gopogh) 74.40 (chart)
Docker_Linux_containerd TestNetworkPlugins/group/bridge/DNS (gopogh) 76.38 (chart)
Docker_Linux TestNetworkPlugins/group/bridge/DNS (gopogh) 84.35 (chart)
Docker_Linux TestNetworkPlugins/group/false/DNS (gopogh) 84.35 (chart)
Docker_Linux TestNetworkPlugins/group/enable-default-cni/DNS (gopogh) 85.22 (chart)
Docker_Linux TestNetworkPlugins/group/kubenet/DNS (gopogh) 86.96 (chart)
Docker_Windows TestNetworkPlugins/group/calico/Start (gopogh) 90.00 (chart)
Docker_Linux TestNetworkPlugins/group/calico/NetCatPod (gopogh) 90.32 (chart)
Docker_Linux_containerd TestNetworkPlugins/group/calico/Start (gopogh) 98.36 (chart)
Docker_macOS TestIngressAddonLegacy/serial/ValidateIngressDNSAddonActivation (gopogh) 98.68 (chart)
Docker_Linux_containerd TestKubernetesUpgrade (gopogh) 100.00 (chart)
Docker_Linux_containerd TestPreload (gopogh) 100.00 (chart)
Docker_macOS TestIngressAddonLegacy/serial/ValidateIngressAddonActivation (gopogh) 100.00 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants