Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

re-establish apiserver tunnel on restart #14183

Merged
merged 4 commits into from
May 17, 2022
Merged

re-establish apiserver tunnel on restart #14183

merged 4 commits into from
May 17, 2022

Conversation

sharifelgamal
Copy link
Collaborator

@sharifelgamal sharifelgamal commented May 17, 2022 •
edited
Loading

After a minikube start --driver=qemu2 and minikube stop:
Before:

😄  minikube v1.26.0-beta.0 on Darwin 12.3.1
✨  Using the qemu2 (experimental) driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing qemu2 VM for "minikube" ...
🐳  Preparing Kubernetes v1.23.6 on Docker 20.10.14 ...
🤦  Unable to restart cluster, will reset it: apiserver health: apiserver healthz never reported healthy: cluster wait timed out during healthz check
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

After:

😄  minikube v1.26.0-beta.0 on Darwin 12.3.1
✨  Using the qemu2 (experimental) driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing qemu2 VM for "minikube" ...
🐳  Preparing Kubernetes v1.23.6 on Docker 20.10.14 ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

This change make tunneling a little smarter, so it knows to tunnel on restart before checking the api server. This prevents the timeout and hard restart we see in the Before section.

Fixes #14171

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 17, 2022
@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 17, 2022
@sharifelgamal
Copy link
Collaborator Author

/ok-to-test

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label May 17, 2022
@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 14183) |
+----------------+----------+---------------------+
| minikube start | 51.1s    | 50.5s               |
| enable ingress | 27.6s    | 26.7s               |
+----------------+----------+---------------------+

Times for minikube start: 52.9s 50.3s 51.0s 50.5s 50.7s
Times for minikube (PR 14183) start: 50.7s 51.0s 49.4s 50.9s 50.7s

Times for minikube ingress: 25.5s 29.5s 29.1s 28.5s 25.6s
Times for minikube (PR 14183) ingress: 29.5s 25.1s 26.5s 26.6s 25.5s

docker driver with docker runtime

+-------------------+----------+---------------------+
|      COMMAND      | MINIKUBE | MINIKUBE (PR 14183) |
+-------------------+----------+---------------------+
| minikube start    | 23.9s    | 24.3s               |
| ⚠️  enable ingress | 22.1s    | 34.3s ⚠️             |
+-------------------+----------+---------------------+

Times for minikube start: 23.8s 23.9s 23.3s 24.7s 23.9s
Times for minikube (PR 14183) start: 24.4s 24.0s 24.7s 24.8s 23.7s

Times for minikube ingress: 21.9s 21.4s 22.9s 22.4s 21.9s
Times for minikube (PR 14183) ingress: 82.4s 21.9s 22.9s 21.9s 22.5s

docker driver with containerd runtime

+-------------------+----------+---------------------+
|      COMMAND      | MINIKUBE | MINIKUBE (PR 14183) |
+-------------------+----------+---------------------+
| ⚠️  minikube start | 29.3s    | 34.9s ⚠️             |
| enable ingress    | 23.7s    | 20.1s               |
+-------------------+----------+---------------------+

Times for minikube start: 29.6s 28.4s 28.0s 27.9s 32.6s
Times for minikube (PR 14183) start: 39.2s 43.5s 31.5s 31.7s 28.5s

Times for minikube ingress: 22.4s 22.4s 22.4s 22.4s 28.9s
Times for minikube (PR 14183) ingress: 22.4s 17.9s 18.9s 18.9s 22.4s

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Docker_Linux_containerd TestNetworkPlugins/group/custom-weave/Start (gopogh) 23.46 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/DeployApp (gopogh) 29.49 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/FirstStart (gopogh) 29.49 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/SecondStart (gopogh) 30.77 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/UserAppExistsAfterStop (gopogh) 34.15 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/DeployApp (gopogh) 35.90 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/FirstStart (gopogh) 36.71 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/UserAppExistsAfterStop (gopogh) 37.18 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/SecondStart (gopogh) 38.75 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/FirstStart (gopogh) 42.31 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/SecondStart (gopogh) 42.31 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/DeployApp (gopogh) 43.04 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/UserAppExistsAfterStop (gopogh) 43.75 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/UserAppExistsAfterStop (gopogh) 55.13 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/DeployApp (gopogh) 56.25 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/FirstStart (gopogh) 56.25 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/SecondStart (gopogh) 56.79 (chart)
Docker_Linux_containerd TestNetworkPlugins/group/kindnet/DNS (gopogh) 57.89 (chart)
Docker_Linux_containerd TestNetworkPlugins/group/calico/Start (gopogh) 71.60 (chart)
Docker_Linux_containerd TestNetworkPlugins/group/enable-default-cni/DNS (gopogh) 76.47 (chart)
Docker_Linux_containerd TestNetworkPlugins/group/bridge/DNS (gopogh) 79.07 (chart)

To see the flake rates of all tests by environment, click here.

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sharifelgamal, spowelljr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [sharifelgamal,spowelljr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 14183) |
+----------------+----------+---------------------+
| minikube start | 51.3s    | 51.1s               |
| enable ingress | 31.3s    | 29.8s               |
+----------------+----------+---------------------+

Times for minikube start: 52.4s 51.4s 50.8s 50.8s 51.0s
Times for minikube (PR 14183) start: 50.5s 51.5s 49.9s 51.9s 51.5s

Times for minikube ingress: 29.6s 30.6s 30.0s 37.2s 29.1s
Times for minikube (PR 14183) ingress: 29.6s 29.1s 31.5s 28.6s 30.1s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 14183) |
+----------------+----------+---------------------+
| minikube start | 24.6s    | 24.7s               |
| enable ingress | 23.3s    | 22.7s               |
+----------------+----------+---------------------+

Times for minikube start: 24.3s 24.7s 24.3s 24.7s 24.9s
Times for minikube (PR 14183) start: 24.8s 24.5s 25.0s 24.8s 24.5s

Times for minikube (PR 14183) ingress: 22.9s 22.9s 21.9s 23.0s 22.9s
Times for minikube ingress: 21.5s 21.4s 23.9s 23.4s 26.4s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 14183) |
+----------------+----------+---------------------+
| minikube start | 35.9s    | 34.4s               |
| enable ingress | 22.4s    | 25.3s               |
+----------------+----------+---------------------+

Times for minikube (PR 14183) start: 29.2s 43.9s 32.1s 39.1s 27.8s
Times for minikube start: 33.6s 33.3s 44.2s 40.2s 28.5s

Times for minikube (PR 14183) ingress: 22.4s 27.9s 20.9s 23.4s 31.9s
Times for minikube ingress: 18.9s 18.9s 18.4s 32.5s 23.4s

@sharifelgamal sharifelgamal merged commit 5748df8 into kubernetes:master May 17, 2022
@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 14183) |
+----------------+----------+---------------------+
| minikube start | 52.1s    | 52.6s               |
| enable ingress | 29.6s    | 30.3s               |
+----------------+----------+---------------------+

Times for minikube start: 51.8s 51.3s 52.2s 52.7s 52.3s
Times for minikube (PR 14183) start: 52.3s 55.3s 51.8s 51.5s 52.3s

Times for minikube (PR 14183) ingress: 30.6s 29.6s 30.1s 30.1s 31.1s
Times for minikube ingress: 30.2s 30.1s 30.1s 28.6s 29.2s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 14183) |
+----------------+----------+---------------------+
| minikube start | 26.4s    | 25.2s               |
| enable ingress | 22.7s    | 22.5s               |
+----------------+----------+---------------------+

Times for minikube start: 29.5s 25.5s 25.4s 25.4s 26.2s
Times for minikube (PR 14183) start: 25.2s 25.0s 25.8s 25.1s 24.6s

Times for minikube ingress: 21.5s 22.5s 22.9s 25.5s 21.0s
Times for minikube (PR 14183) ingress: 23.0s 24.5s 22.5s 21.9s 20.4s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 14183) |
+----------------+----------+---------------------+
| minikube start | 32.7s    | 30.2s               |
| enable ingress | 21.4s    | 23.5s               |
+----------------+----------+---------------------+

Times for minikube (PR 14183) start: 32.2s 30.1s 29.8s 29.5s 29.6s
Times for minikube start: 34.8s 33.8s 31.9s 30.0s 33.0s

Times for minikube ingress: 19.0s 18.0s 29.0s 22.0s 19.0s
Times for minikube (PR 14183) ingress: 18.9s 22.4s 22.0s 21.9s 32.4s

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
KVM_Linux_containerd TestStoppedBinaryUpgrade/Upgrade (gopogh) 1.75 (chart)
Hyper-V_Windows TestNetworkPlugins/group/enable-default-cni/Start (gopogh) 13.04 (chart)
Hyper-V_Windows TestNetworkPlugins/group/flannel/Start (gopogh) 21.74 (chart)
Docker_Linux_containerd TestNetworkPlugins/group/custom-weave/Start (gopogh) 23.46 (chart)
Hyper-V_Windows TestNetworkPlugins/group/bridge/Start (gopogh) 36.96 (chart)
Hyper-V_Windows TestNetworkPlugins/group/kubenet/Start (gopogh) 44.57 (chart)
Hyper-V_Windows TestMultiNode/serial/PingHostFrom2Pods (gopogh) 100.00 (chart)
Hyper-V_Windows TestMultiNode/serial/RestartKeepsNodes (gopogh) 100.00 (chart)
Hyper-V_Windows TestNoKubernetes/serial/StartWithStopK8s (gopogh) 100.00 (chart)
Hyper-V_Windows TestSkaffold (gopogh) 100.00 (chart)

To see the flake rates of all tests by environment, click here.

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Hyperkit_macOS TestForceSystemdFlag (gopogh) 0.00 (chart)
Docker_Windows TestFunctional/parallel/TunnelCmd/serial/WaitService/IngressIP (gopogh) 11.11 (chart)
Docker_Linux_containerd TestNetworkPlugins/group/custom-weave/Start (gopogh) 23.46 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/DeployApp (gopogh) 29.49 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/FirstStart (gopogh) 29.49 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/SecondStart (gopogh) 30.77 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/UserAppExistsAfterStop (gopogh) 34.15 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/DeployApp (gopogh) 35.90 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/FirstStart (gopogh) 36.71 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/UserAppExistsAfterStop (gopogh) 37.18 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/SecondStart (gopogh) 38.75 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/FirstStart (gopogh) 42.31 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/SecondStart (gopogh) 42.31 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/DeployApp (gopogh) 43.04 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/UserAppExistsAfterStop (gopogh) 43.75 (chart)
Docker_Windows TestCertOptions (gopogh) 49.52 (chart)
Docker_Windows TestStartStop/group/old-k8s-version/serial/AddonExistsAfterStop (gopogh) 51.92 (chart)
Docker_Windows TestStartStop/group/old-k8s-version/serial/SecondStart (gopogh) 51.92 (chart)
Docker_Windows TestStartStop/group/old-k8s-version/serial/UserAppExistsAfterStop (gopogh) 51.92 (chart)
Docker_Windows TestStartStop/group/old-k8s-version/serial/VerifyKubernetesImages (gopogh) 51.92 (chart)
Docker_Windows TestForceSystemdEnv (gopogh) 54.29 (chart)
Docker_Windows TestNetworkPlugins/group/kubenet/Start (gopogh) 54.29 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/UserAppExistsAfterStop (gopogh) 55.13 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/DeployApp (gopogh) 56.25 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/FirstStart (gopogh) 56.25 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/SecondStart (gopogh) 56.79 (chart)
Docker_Linux_containerd TestNetworkPlugins/group/kindnet/DNS (gopogh) 57.89 (chart)
Docker_Windows TestNoKubernetes/serial/Start (gopogh) 58.25 (chart)
Docker_Windows TestNetworkPlugins/group/bridge/Start (gopogh) 59.05 (chart)
Docker_Windows TestNetworkPlugins/group/kindnet/Start (gopogh) 65.71 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@spowelljr spowelljr spowelljr approved these changes

@blueelvis blueelvis Awaiting requested review from blueelvis

@medyagh medyagh Awaiting requested review from medyagh

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Qemu2 driver, Can not Start after a stop
4 participants
@sharifelgamal @minikube-pr-bot @k8s-ci-robot @spowelljr