Cross-dock tests are flaky

[jpkrohling]

I'm noticing that master is marked as red in Travis quite often. Whenever I notice it, I just start again the test that failed, and the culprit is almost always cross-dock, with failures like:

Executing Matrix...
✗ [endtoend] test_driver→ (services=go) ⇒ Fail: could not retrieve traces from query service
✓ [endtoend] test_driver→ (services=java) ⇒ Pass
✓ [endtoend] test_driver→ (services=node) ⇒ Pass
✓ [endtoend] test_driver→ (services=python) ⇒ Pass
✓ [endtoend] test_driver→ (services=zipkin-brave-json) ⇒ Pass
✓ [endtoend] test_driver→ (services=zipkin-brave-json-v2) ⇒ Pass
✓ [endtoend] test_driver→ (services=zipkin-brave-thrift) ⇒ Pass
6/7 passed (0/7 skipped)
Tests did not pass!
2019/04/02 14:20:48 At least one test failed
grep 'Tests passed!' run-crossdock.log
make[1]: *** [crossdock] Error 1
make[1]: Leaving directory `/home/travis/gopath/src/github.com/jaegertracing/jaeger'
make: *** [build-crossdock] Error 2
The command "if [ "$CROSSDOCK" == true ]; then bash ./scripts/travis/build-crossdock.sh ; else echo 'skipping crossdock'; fi" exited with 2.

[yurishkuro]

I tried to increase logging recently, but it didn't help much, so still don't know what's causing it, only know that both times this started happening in the past 6mo was strongly correlated with enabling grpc instead of tchannel (non-scientific evidence).

[jpkrohling]

I'll reuse this issue to track the otel-crossdock tests, as they are also flaky.

[pavolloffay]

outdatted