Heartbeat Telemetry Test is Flakey

[andrewvc]

Flaky Test

• Test Name: test_telemetry.Test.test_telemetry (from nosetests)
• Link: https://github.com/elastic/beats/blob/master/heartbeat/tests/system/test_telemetry.py#L14
• Artifact Link: archive (2).zip
• Notes: This consistently works locally but often fails on CI

Stack Trace

Error Message
Timeout waiting for 'cond' to be true. Waited 10 seconds.
Stacktrace
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/private/var/lib/jenkins/workspace/elastic+beats+pull-request+multijob-darwin/beat/heartbeat/label/macosx/src/github.com/elastic/beats/heartbeat/tests/system/test_telemetry.py", line 62, in test_telemetry
    self.wait_until(lambda: self.output_has(lines=init_lines+2))
  File "/private/var/lib/jenkins/workspace/elastic+beats+pull-request+multijob-darwin/beat/heartbeat/label/macosx/src/github.com/elastic/beats/heartbeat/tests/system/../../../libbeat/tests/system/beat/beat.py", line 337, in wait_until
    "Waited {} seconds.".format(max_timeout))
TimeoutError: Timeout waiting for 'cond' to be true. Waited 10 seconds.
Standard Error
127.0.0.1 - - [07/Nov/2018 05:24:13] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [07/Nov/2018 05:24:14] "GET / HTTP/1.1" 200 -

[elasticmachine]

Pinging @elastic/uptime

[jlind23]

Backlog grooming: Closing it for now until further activity, can still be reopened if needed.

[ruflin]

I'm a bit worried that we close flaky test issues. I think we need to keep these open as we should treat it like bugs.

[jlind23]

@ruflin flakky test issues opened 3 years ago are either not important or not relevant.
It don't see any added value at keeping them opened.

[ruflin]

The problem is that we kept them open for 3 years ;-) @andrewvc

[andrewvc]

Glad to see this closed, haven't noticed any issues here for... years?

[ruflin]

@andrewvc Does the test still exist and is skipped? If yes, lets remove it instead of skipping it.