[target-allocator] restart pod watcher when no event is found

[moh-osman3]

Resolves #1028

Before fix:

{"level":"info","ts":1668017421.9970064,"msg":"Starting the Target Allocator"}
{"level":"info","ts":1668017421.9972935,"logger":"allocator","msg":"Unrecognized filter strategy; filtering disabled"}
{"level":"info","ts":1668017422.0179584,"logger":"setup","msg":"Starting server..."}
{"level":"info","ts":1668017422.0202508,"logger":"allocator","msg":"Successfully started a collector pod watcher","component":"opentelemetry-targetallocator"}
{"level":"info","ts":1665127338.1837273,"logger":"allocator","msg":"false","component":"opentelemetry-targetallocator"}
{"level":"info","ts":1665127338.1838017,"logger":"allocator","msg":"Collector pod watch event stopped no event","component":"opentelemetry-targetallocator"}

This meant pod watcher is no longer running and the TA assigns remaining targets to pods that were already terminated

After fix:

{"level":"info","ts":1668017421.9970064,"msg":"Starting the Target Allocator"}
{"level":"info","ts":1668017421.9972935,"logger":"allocator","msg":"Unrecognized filter strategy; filtering disabled"}
{"level":"info","ts":1668017422.0179584,"logger":"setup","msg":"Starting server..."}
{"level":"info","ts":1668017422.0202508,"logger":"allocator","msg":"Successfully started a collector pod watcher","component":"opentelemetry-targetallocator"}
{"level":"info","ts":1668020290.9531963,"logger":"allocator","msg":"No event found. Restarting watch routine","component":"opentelemetry-targetallocator"}
{"level":"info","ts":1668020290.9575274,"logger":"allocator","msg":"Successfully started a collector pod watcher","component":"opentelemetry-targetallocator"}
{"level":"info","ts":1668023632.289222,"logger":"allocator","msg":"No event found. Restarting watch routine","component":"opentelemetry-targetallocator"}
{"level":"info","ts":1668023632.292879,"logger":"allocator","msg":"Successfully started a collector pod watcher","component":"opentelemetry-targetallocator"}

pod watcher gets restarted until stabilization occurs -> confirmed TA is assigning targets to correct remaining pods

[Aneurysm9]

Is this going to accumulate deferred executions for every iteration through this infinite loop? Do we need to defer execution here? Can it simply be not executed or can it be executed without deferral at the end of each loop iteration?

[moh-osman3]

Thanks for the review! Hmm I think if the operation returns before the timeout (no timeout triggered) then a call to cancel() is needed for cleanup. I agree I shouldn't pile up a bunch of defer statements and should call cancel() as soon as possible. If I add cancel() to the end of for loop and before the returns, I think there's still a chance if a panic occurs cancel() won't be called without defer.

To solve this I wrapped this whole block into its own method called restartWatch and that way defer can be called properly. Let me know if you have any thoughts on this!

[kristinapathak]

Other than the logging change (which can also be done in a separate PR if we want), this looks awesome!