Prevent deadlock in TestDynamicEnqueueRequest

[david-kow]

This PR introduces a preventive measure to stop TestDynamicEnqueueRequest from a deadlock. It happens because, contrary to our expectations, we can receive more than one event even though we only do a single update after registering our handler. In my tests I saw we can receive two or even three updates. Since the channel we use is unbuffered and we do a single read, our handler is blocked. As staying in the handler prevents from shutting down gracefully, we timeout after 30 seconds with an error and fail the test.

The change makes the handler respect the provided context, allowing controller-runtime to shut down gracefully.

I've also randomized the name to facilitate running the test multiple times, as it seems the teardown is not thorough and resources with same name clash.

Before the change (although with random resource name) I was getting around 1% failure rate every time on the below:

go test -v -race -count 1000 -tags=integration pkg/controller/common/watches/handler_integration_test.go

After the change running (note 3k count):

go test -v -race -count 3000 -tags=integration pkg/controller/common/watches/handler_integration_test.go

Passes without any failures.

Fixes #4692.

[thbkrkr]

Well seen! When I ran the test command with -race and -count I lacked ambition with only -count 100. With -count 1000, I've failures "which is great" but a lower rate failure than you: 0.1% or 0.2%.

LGTM!