Cherry-pick #21851 to 7.x: Refactor docker watcher to fix flaky test and other small issues

[jsoriano]

Cherry-pick of PR #21851 to 7.x branch. Original message:

What does this PR do?

Refactor docker watcher to fix some small issues and improve testability:

• Actually release resources of previous connections when reconnecting.
• Watcher uses a clock that can be mocked in tests for time-sensitive functionality.
• Use nanoseconds-precision from events timestamps, this is important to avoid duplicated events on reconnection.
• Fix logger initialization (it was being initialized as docker.docker).
• Refactor test helpers to have more control on test watcher when needed.
• Some other code refactors.

Why is it important?

• Fixes flaky test [Libbeat] Flaky TestWatcherDie test #7906.
• Watch loop relied on deferred calls to do cleanups, but the watch loop never finished, so resources associated to each reconnection were never released. On reconnections caused by not receiving events, previous connections were not being closed, so several connections were kept alive, but all except the last one were ignored.
• Fix duplication of events on reconnections. Each reconnection asks for events since last received events, using seconds granularity made each reconnection to retrieve again the events happened during the second the last event was received, including the event itself. API supports nanosecond granularity, that avoids this problem.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Fixes [Libbeat] Flaky TestWatcherDie test #7906.

$ go test -count 10000 ./libbeat/common/docker/
ok  	github.com/elastic/beats/v7/libbeat/common/docker	5.220s

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: [Pull request #21918 opened]

• Start Time: 2020-10-16T15:57:49.572+0000

• Duration: 108 min 12 sec

Test stats 🧪

Test	Results
Failed	0
Passed	16338
Skipped	1364
Total	17702