Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

multinode: fix control plane not ready on restart #8698

Merged
merged 8 commits into from
Jul 16, 2020
Merged

multinode: fix control plane not ready on restart #8698

merged 8 commits into from
Jul 16, 2020

Conversation

sharifelgamal
Copy link
Collaborator

Fixes #8600

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jul 10, 2020
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sharifelgamal

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 10, 2020
@medyagh medyagh changed the title update control plane properly on multinode restart multinode: fix control plane not ready on restart Jul 10, 2020
@medyagh
Copy link
Member

medyagh commented Jul 10, 2020

/ok-to-test

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Jul 10, 2020
@minikube-pr-bot
Copy link

kvm2 Driver
docker Driver

@medyagh
Copy link
Member

medyagh commented Jul 11, 2020

Restart mutlinode test failed in gh action and jenkins

@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jul 13, 2020
@minikube-pr-bot
Copy link

kvm2 Driver
Times for minikube: [63.395773223999996 62.948747802999996 62.95453990000001]
Average time for minikube: 63.099686975666664

Times for Minikube (PR 8698): [66.956771815 61.817401701 67.55278709000001]
Average time for Minikube (PR 8698): 65.442320202

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 8698) |
+--------------------------------+-----------+--------------------+
| * minikube v1.12.0 on Debian   |  0.061149 |           0.058207 |
|                           9.11 |           |                    |
| * Using the kvm2 driver based  |  0.019940 |           0.020221 |
| on existing profile            |           |                    |
| * Starting control plane node  |  0.018578 |           0.011582 |
| minikube in cluster minikube   |           |                    |
| * Creating kvm2 VM (CPUs=2,    | 40.687434 |          39.753913 |
| Memory=3700MB, Disk=20000MB)   |           |                    |
| ...                            |           |                    |
| * Preparing Kubernetes v1.18.3 | 20.716186 |          23.716130 |
| on Docker 19.03.12 ...         |           |                    |
| * Verifying Kubernetes         |  1.290910 |           1.462192 |
| components...                  |           |                    |
| * Enabled addons:              |  0.216773 |           0.342546 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.084756 |           0.072508 |
| configured to use "minikube"   |           |                    |
|                                |  0.003961 |           0.005021 |
+--------------------------------+-----------+--------------------+

docker Driver
Times for minikube: [27.354459246000005 26.260922277000006 24.684449708999995]
Average time for minikube: 26.099943744

Times for Minikube (PR 8698): [26.068001815 26.878878755000002 25.971940163000003]
Average time for Minikube (PR 8698): 26.30627357766667

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 8698) |
+--------------------------------+-----------+--------------------+
| * minikube v1.12.0 on Debian   |  0.073177 |           0.070752 |
|                           9.11 |           |                    |
| * Using the docker driver      |  0.002761 |           0.002433 |
| based on existing profile      |           |                    |
| * Starting control plane node  |  0.057940 |           0.058091 |
| minikube in cluster minikube   |           |                    |
| * Creating docker container    |  7.714459 |           7.489948 |
| (CPUs=2, Memory=3700MB) ...    |           |                    |
| * Preparing Kubernetes v1.18.3 | 17.316825 |          17.685894 |
| on Docker 19.03.2 ...          |           |                    |
| * Verifying Kubernetes         |  0.863298 |           0.910828 |
| components...                  |           |                    |
| * Enabled addons:              |  0.002935 |           0.019278 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.064496 |           0.064479 |
| configured to use "minikube"   |           |                    |
|                                |  0.004054 |           0.004570 |
+--------------------------------+-----------+--------------------+

@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 14, 2020
@minikube-pr-bot
Copy link

kvm2 Driver
Times for minikube: [66.747257383 58.394459907000005 61.846409698]
Average time for minikube: 62.32937566266667

Times for Minikube (PR 8698): [64.426172565 63.275204669 63.86242006]
Average time for Minikube (PR 8698): 63.854599097999994

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 8698) |
+--------------------------------+-----------+--------------------+
| * minikube v1.12.0 on Debian   |  0.060890 |           0.058120 |
|                           9.11 |           |                    |
| * Using the kvm2 driver based  |  0.020402 |           0.019504 |
| on existing profile            |           |                    |
| * Starting control plane node  |  0.009148 |           0.007484 |
| minikube in cluster minikube   |           |                    |
| * Creating kvm2 VM (CPUs=2,    | 38.847625 |          40.857418 |
| Memory=3700MB, Disk=20000MB)   |           |                    |
| ...                            |           |                    |
| * Preparing Kubernetes v1.18.3 | 21.462387 |          21.569621 |
| on Docker 19.03.12 ...         |           |                    |
| * Verifying Kubernetes         |  1.293836 |           1.208418 |
| components...                  |           |                    |
| * Enabled addons:              |  0.539397 |           0.060953 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.091944 |           0.069607 |
| configured to use "minikube"   |           |                    |
|                                |  0.003747 |           0.003473 |
+--------------------------------+-----------+--------------------+

docker Driver
Times for minikube: [26.366106334000005 26.736700506000002 25.027741804]
Average time for minikube: 26.04351621466667

Times for Minikube (PR 8698): [25.670679615999997 25.058768182 26.182394657]
Average time for Minikube (PR 8698): 25.637280818333334

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 8698) |
+--------------------------------+-----------+--------------------+
| * minikube v1.12.0 on Debian   |  0.070244 |           0.072590 |
|                           9.11 |           |                    |
| * Using the docker driver      |  0.002919 |           0.002618 |
| based on existing profile      |           |                    |
| * Starting control plane node  |  0.055386 |           0.055818 |
| minikube in cluster minikube   |           |                    |
| * Creating docker container    |  7.552148 |           7.529301 |
| (CPUs=2, Memory=3700MB) ...    |           |                    |
| * Preparing Kubernetes v1.18.3 | 17.362146 |          16.977670 |
| on Docker 19.03.2 ...          |           |                    |
| * Verifying Kubernetes         |  0.901628 |           0.847624 |
| components...                  |           |                    |
| * Enabled addons:              |  0.031435 |           0.082176 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.062622 |           0.064098 |
| configured to use "minikube"   |           |                    |
|                                |  0.004990 |           0.005386 |
+--------------------------------+-----------+--------------------+

@TravisBuddy
Copy link

Travis tests have failed

Hey @sharifelgamal,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.

1st Build

View build log

make test 
which go-bindata || GO111MODULE=off GOBIN="/home/travis/gopath/bin" go get github.com/jteeuwen/go-bindata/...
PATH="/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.14.4.linux.amd64/bin:/home/travis/bin:/home/travis/bin:/home/travis/.local/bin:/usr/local/lib/jvm/openjdk11/bin:/opt/pyenv/shims:/home/travis/.phpenv/shims:/home/travis/perl5/perlbrew/bin:/home/travis/.nvm/versions/node/v8.12.0/bin:/home/travis/.rvm/gems/ruby-2.5.3/bin:/home/travis/.rvm/gems/ruby-2.5.3@global/bin:/home/travis/.rvm/rubies/ruby-2.5.3/bin:/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.11.1.linux.amd64/bin:/usr/local/maven-3.6.0/bin:/usr/local/cmake-3.12.4/bin:/usr/local/clang-7.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/home/travis/.rvm/bin:/home/travis/.phpenv/bin:/opt/pyenv/bin:/home/travis/.yarn/bin:/home/travis/gopath/bin" go-bindata -nomemcopy -o pkg/minikube/assets/assets.go -pkg assets deploy/addons/...
gofmt -s -w pkg/minikube/assets/assets.go
which go-bindata || GO111MODULE=off GOBIN="/home/travis/gopath/bin" go get github.com/jteeuwen/go-bindata/...
/home/travis/gopath/bin/go-bindata
PATH="/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.14.4.linux.amd64/bin:/home/travis/bin:/home/travis/bin:/home/travis/.local/bin:/usr/local/lib/jvm/openjdk11/bin:/opt/pyenv/shims:/home/travis/.phpenv/shims:/home/travis/perl5/perlbrew/bin:/home/travis/.nvm/versions/node/v8.12.0/bin:/home/travis/.rvm/gems/ruby-2.5.3/bin:/home/travis/.rvm/gems/ruby-2.5.3@global/bin:/home/travis/.rvm/rubies/ruby-2.5.3/bin:/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.11.1.linux.amd64/bin:/usr/local/maven-3.6.0/bin:/usr/local/cmake-3.12.4/bin:/usr/local/clang-7.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/home/travis/.rvm/bin:/home/travis/.phpenv/bin:/opt/pyenv/bin:/home/travis/.yarn/bin:/home/travis/gopath/bin" go-bindata -nomemcopy -o pkg/minikube/translate/translations.go -pkg translate translations/...
gofmt -s -w pkg/minikube/translate/translations.go
MINIKUBE_LDFLAGS="-X k8s.io/minikube/pkg/version.version=v1.12.0 -X k8s.io/minikube/pkg/version.isoVersion=v1.12.0 -X k8s.io/minikube/pkg/version.isoPath=minikube/iso -X k8s.io/minikube/pkg/version.gitCommitID="2f52099048ac7f70d9243aa549fb8dd1c0af4015"" ./test.sh
= make lint =============================================================
golangci/golangci-lint info checking GitHub for tag 'v1.26.0'
golangci/golangci-lint info found version: 1.26.0 for v1.26.0/linux/amd64
golangci/golangci-lint info installed out/linters/golangci-lint
test/integration/multinode_test.go:245:6: ineffectual assignment to `err` (ineffassign)
	rr, err = Run(t, exec.CommandContext(ctx, "kubectl", "get", "nodes"))
	    ^
test/integration/multinode_test.go:250:6: ineffectual assignment to `err` (ineffassign)
	rr, err = Run(t, exec.CommandContext(ctx, "kubectl", "get", "nodes", "-o", `go-template='{{range .items}}{{range .status.conditions}}{{if eq .type "Ready"}} {{.status}}{{"\n"}}{{end}}{{end}}{{end}}'`))
	    ^
Makefile:390: recipe for target 'lint-ci' failed
make[1]: *** [lint-ci] Error 1
= go mod ================================================================
ok
= boilerplate ===========================================================
ok
Makefile:274: recipe for target 'test' failed
make: *** [test] Error 4
TravisBuddy Request Identifier: c601e760-c6e7-11ea-aa06-e17841301e13

@minikube-pr-bot
Copy link

kvm2 Driver
Times for minikube: [63.560546136 65.823352181 67.455718995]
Average time for minikube: 65.61320577066665

Times for Minikube (PR 8698): [62.60711526800001 68.28837775599999 63.468774929]
Average time for Minikube (PR 8698): 64.78808931766666

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 8698) |
+--------------------------------+-----------+--------------------+
| * minikube v1.12.0 on Debian   |  0.063931 |           0.065394 |
|                           9.11 |           |                    |
| * Using the kvm2 driver based  |  0.020236 |           0.022005 |
| on existing profile            |           |                    |
| * Starting control plane node  |  0.005029 |           0.003377 |
| minikube in cluster minikube   |           |                    |
| * Creating kvm2 VM (CPUs=2,    | 40.870193 |          41.192600 |
| Memory=3700MB, Disk=20000MB)   |           |                    |
| ...                            |           |                    |
| * Preparing Kubernetes v1.18.3 | 22.616917 |          21.811290 |
| on Docker 19.03.12 ...         |           |                    |
| * Verifying Kubernetes         |  1.393783 |           1.480006 |
| components...                  |           |                    |
| * Enabled addons:              |  0.561050 |           0.134758 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.078922 |           0.074132 |
| configured to use "minikube"   |           |                    |
|                                |  0.003144 |           0.004529 |
+--------------------------------+-----------+--------------------+

docker Driver
Times for minikube: [29.905466684 26.331447788 27.557172334999997]
Average time for minikube: 27.931362269

Times for Minikube (PR 8698): [26.868504083999998 27.713833857 26.800029612]
Average time for Minikube (PR 8698): 27.127455850999997

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 8698) |
+--------------------------------+-----------+--------------------+
| * minikube v1.12.0 on Debian   |  0.078444 |           0.082314 |
|                           9.11 |           |                    |
| * Using the docker driver      |  0.002686 |           0.002556 |
| based on existing profile      |           |                    |
| * Starting control plane node  |  0.063158 |           0.063104 |
| minikube in cluster minikube   |           |                    |
| * Creating docker container    |  7.964612 |           7.707685 |
| (CPUs=2, Memory=3700MB) ...    |           |                    |
| * Preparing Kubernetes v1.18.3 | 18.775553 |          17.991636 |
| on Docker 19.03.2 ...          |           |                    |
| * Verifying Kubernetes         |  0.965697 |           1.175617 |
| components...                  |           |                    |
| * Enabled addons:              |  0.012315 |           0.003120 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.064707 |           0.090518 |
| configured to use "minikube"   |           |                    |
|                                |  0.004190 |           0.010905 |
+--------------------------------+-----------+--------------------+

@sharifelgamal sharifelgamal merged commit 97484a2 into kubernetes:master Jul 16, 2020
@fmotrifork fmotrifork mentioned this pull request Aug 3, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@medyagh medyagh medyagh approved these changes

@afbjorklund afbjorklund Awaiting requested review from afbjorklund

@prasadkatti prasadkatti Awaiting requested review from prasadkatti

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

multinode: minikube cluster NotReady after restart
5 participants
@sharifelgamal @k8s-ci-robot @medyagh @minikube-pr-bot @TravisBuddy