Flake Test: TestMultiNode/serial/RestartMultiNode: node X has unwanted condition "Ready" : Reason "NodeStatusUnknown" Message: "Kubelet stopped posting node status.". will try.

[medyagh]

Example https://storage.googleapis.com/minikube-builds/logs/10638/8b10177/Docker_Linux.html#fail_TestMultiNode%2fserial%2fRestartMultiNode

	I0226 19:02:29.778088   83784 node_ready.go:53] node "multinode-20210226185242-6625-m02" has unwanted condition "Ready" : Reason "NodeStatusUnknown" Message: "Kubelet stopped posting node status.". will try. 
	I0226 19:02:29.781519   83784 node_ready.go:53] node "multinode-20210226185242-6625-m02" has unwanted condition "Ready" : Reason "NodeStatusUnknown" Message: "Kubelet stopped posting node status.". will try. 
	I0226 19:02:29.781569   83784 node_ready.go:38] duration metric: took 4m0.012501852s to wait for WaitForNodeReady...
	I0226 19:02:29.784428   83784 out.go:119] 
	W0226 19:02:29.784579   83784 out.go:181] X Exiting due to GUEST_START: wait 6m0s for node: waiting for node to be ready: wait node ready: timed out waiting for the condition
	X Exiting due to GUEST_START: wait 6m0s for node: waiting for node to be ready: wait node ready: timed out waiting for the condition
	W0226 19:02:29.784629   83784 out.go:181] * 
	* 
	W0226 19:02:29.784676   83784 out.go:181] * If the above advice does not help, please let us know: 
	* If the above advice does not help, please let us know: 
	W0226 19:02:29.784723   83784 out.go:181]   - https://github.com/kubernetes/minikube/issues/new/choose
	  - https://github.com/kubernetes/minikube/issues/new/choose
	I0226 19:02:29.787189   83784 out.go:119] 
** /stderr **

[jiefenghe]

/assign

[medyagh]

thanks @jiefenghe for talking this, please let me know if u have any questions
@prezha might have some insight on this, please feel free to brainstorm togheter

[ilya-zuyev]

It looks like the tests started to fail on Feb 16

[ilya-zuyev]

@prezha please take a look at https://storage.googleapis.com/minikube-builds/logs/10424/90cd9c3/Docker_Linux.html looks

[medyagh]

	t.Run("serial", func(t *testing.T) {
		tests := []struct {
			name      string
			validator validatorFunc
		}{
			{"FreshStart2Nodes", validateMultiNodeStart},
			{"AddNode", validateAddNodeToMultiNode},
			{"ProfileList", validateProfileListWithMultiNode},
			{"StopNode", validateStopRunningNode},
			{"StartAfterStop", validateStartNodeAfterStop},
			{"DeleteNode", validateDeleteNodeFromMultiNode},
			{"StopMultiNode", validateStopMultiNodeCluster},
			{"RestartMultiNode", validateRestartMultiNodeCluster},
			{"ValidateNameConflict", validatNameConflict},
		}

so it happens after StopMultiNode
and then Starting it again
but not on a Normal FreshStart2Nodes

[prezha]

@jiefenghe i've created pr #10758 describing what i've found and proposing solution, please have a look and share your thoughts