Fix pluginwatcher flake

[vikaschoudhary16]

This is a stopgap(temporary) fix for the flake, #20136
This part has got changed recently, and this issue is not there at upstream kube. This PR,kubernetes/kubernetes#65662, is cherry-picking the changed part to kube 1.11.

Meanwhile this fix will help avoiding this flake.

Implementation of NotifyRegistrationStatus RPC at example plugin is writing registration status to a channel that unit test is listening on. Unit test after receving data on this channel, stops the example plugin server. Here is a race. Sometimes plugin server gets closed before NotifyRegistrationStatus could return and thus showing error on client (plugin watcher), which if being verified in the unit test.

Fix: With this 1 ms delay, plugin server is not getting closed before RPC return.

WIthout this fix, issue is happening at my local machine with this cmd:

go test -race -count 10 k8s.io/kubernetes/pkg/kubelet/util/pluginwatcher -run TestExamplePlugin

After this fix, following is also working:

go test -race -count 500 k8s.io/kubernetes/pkg/kubelet/util/pluginwatcher -run TestExamplePlugin

UPDATE: Increased sleep time to 1 sec

/cc @sjenning @liggitt

[liggitt]

This is still flake prone at load. If a sleep is needed, sleep for longer.

[vikaschoudhary16]

increased to 1 sec, i think it should be enough.

[sjenning]

@vikaschoudhary16 please make the commit msg UPSTREAM: <drop>: Fix pluginwatcher flake so that it will pass hack/verify-upstream-commits.sh and be dropped on the next rebase.

[sjenning]

/lgtm

[deads2k]

/approve

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, sjenning, vikaschoudhary16

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• vendor/k8s.io/kubernetes/pkg/kubelet/OWNERS [deads2k]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[smarterclayton]

Package is now flaking with:

=== RUN   TestExamplePlugin
E0706 05:58:32.056187   21679 example_plugin.go:129] example server stopped serving: accept unix /tmp/plugin_test285297923/plugin.sock: use of closed network connection
E0706 05:58:33.056929   21679 example_plugin.go:147] Timed out on waiting for stop completion
--- FAIL: TestExamplePlugin (2.20s)
	assertions.go:254: 
			Error Trace:	plugin_watcher_test.go:138
			Error:      	Received unexpected error:
			            	Timed out on waiting for stop completion
			Test:       	TestExamplePlugin

in https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/20231/pull-ci-origin-unit/965/

[vikaschoudhary16]

Sorry, my mistake. Did not retest after increasing timeout from 1 ms to 1 sec. Missed to notice that timeout duration for plugin stop is also 1 sec and therefore it started flaking, which is obvious.
Fixing this mistake here: #20237