Fix deadlock when Telegraf is aligning aggregators

[adrianlzt]

If a Telegraf is asked to stop (SIGTERM, SIGINT) while it is aligning an
aggregator, internal.SleepContext will fail and the goroutine will
return without closing the aggregations channel getting stuck
forever in for metric := range aggregations

Required for all PRs:

• Signed CLA.
• Associated README.md updated.
• Has appropriate unit tests.

Sample config file to test the problem:

[global_tags]
[agent]
  interval = "60s"
  debug = true

[[aggregators.basicstats]]
  period = "1800s"

[[inputs.mem]]

[[outputs.file]]
  files = [ "stdout" ]

Run telegraf:

telegraf -config telegraf.conf

Kill it before 60s:

pkill telegraf

It won't finish.

Here telegraf.log is the log of the execution mixed with the kill -ABRT used to check the status of goroutines.

goroutine 58 (agent.runAggregators) is the problematic one. It is stuck so it is not closing the channel used by the outputs.

goroutine 59 (agent.runOutputs) is waiting till the aggregator close the input channel.

goroutine 60 (agent.flush): this one should be stopped by runOutputs

goroutine 36 (ticker) is going to stop when agent.flushOnce finishes. Meanwhile is keeping one goroutine running so Go could not detect a deadlock.

goroutine 51 (opencensus) is strange. I have not configured nothing related with opencensus. Looking at the code, that lib have a init() function that starts its own goroutine with a time Ticker. At first glance do not look like a good pattern to start a goroutine in the init() function, in fact using the init() looks like is not a good pattern.
Removing this import does not solve the not-detected-deadlock problem.
Maybe could be a good idea to check for no init() functions in the vendor libs, but maybe too much.

[danielnelson]

Thanks for the fix and also for opening the issue with opencensus-go, I agree this should be a requirement going forward. We also need to be careful not to bring in any libraries that do excessive static initialization as well.

[danielnelson]

I should clarify that I think using init() is fine, but we shouldn't bring in libraries that start goroutines as a side effect of importing them and we need to be careful about libraries that are initializing large amounts of data.