TestDynamicEnqueueRequest is flaky

[barkbay]

TestDynamicEnqueueRequest is flaky.

=== FAIL: pkg/controller/common/watches TestDynamicEnqueueRequest (30.20s)
{
  "log.level":"info","@timestamp":"2021-10-12T12:48:01.129Z","log.logger":"controller-runtime.manager.controller.test-reconciler",
  "message":"Starting EventSource","service.version":"0.0.0-SNAPSHOT+00000000","service.type":"eck","ecs.version":"1.4.0","source":"kind source: /, Kind="}
{
  "log.level":"info","@timestamp":"2021-10-12T12:48:01.130Z","log.logger":"controller-runtime.manager.controller.test-reconciler",
  "message":"Starting Controller","service.version":"0.0.0-SNAPSHOT+00000000","service.type":"eck","ecs.version":"1.4.0"}
{
  "log.level":"info","@timestamp":"2021-10-12T12:48:01.232Z","log.logger":"controller-runtime.manager.controller.test-reconciler",
  "message":"Starting workers","service.version":"0.0.0-SNAPSHOT+00000000","service.type":"eck","ecs.version":"1.4.0","worker count":1}
{
  "log.level":"debug","@timestamp":"2021-10-12T12:48:01.239Z","log.logger":"dynamic-enqueue-request",
  "message":"Adding new handler registration","service.version":"0.0.0-SNAPSHOT+00000000","service.type":"eck","ecs.version":"1.4.0","key":"test-watch-1","current_registrations":{}}
{
  "log.level":"info","@timestamp":"2021-10-12T12:48:01.246Z","log.logger":"controller-runtime.manager.controller.test-reconciler",
  "message":"Shutdown signal received, waiting for all workers to finish","service.version":"0.0.0-SNAPSHOT+00000000","service.type":"eck","ecs.version":"1.4.0"}

integration.go:97: 
	Error Trace:	integration.go:97
        				handler_integration_test.go:82
	Error:      	Received unexpected error:
	            	failed waiting for all runnables to end within grace period of 30s: context deadline exceeded
	Test:       	TestDynamicEnqueueRequest

-> controller-runtime source code that returns this error (pkg/manager/internal.go)

It has been first noticed by @thbkrkr while we were upgrading controller-runtime to v0.9.1. It also failed a couple of times for these PRs:

• Bump go.elastic.co/apm from 1.8.0 to 1.12.0 #4689
• Add v1alpha1/[es|kb|apm] versions in v1 crds (#4679) #4681
• OperatorHub generator: ensure YAML sep. between documents (#4666) #4667
• [Stack Monitoring] Kibana monitoring with Metricbeat and Filebeat as sidecars #4618 (here)
• Refactor Elasticsearch HTTP error handling #4922
• Skip remote clusters reconciliation earlier if remote clusters are not configured #4924
• Add operator license key check #4925
• Fix 8.0.0 related issues in E2E #4931
• Shorten the reconciliation loop duration if Elasticsearch is down #4938
• Fix Fleet Kubernetes recipe and ES auth E2E failures #4944

[thbkrkr]

I haven't found a way to reproduce the problem yet. My intuition would say that there is a deadlock somewhere that prevents the stop from ending but this is pure speculation.

I don't have a lot of ideas to mitigate this issue other than:

• Completely remove the test as it doesn't test much, in particular since Transform dynamic watches integration tests into unit tests #911
• Increase the gracefulShutdownTimeout duration, though I don't think it will change something
• Do not fail the integration test when we get the error failed waiting for all runnables to end within grace period

Any other ideas?