Wallaroo cluster intermittently stalls and never finishes processing in CI tests

[nisanharamati]

Integration and Integration-with-resilience tests fail sporadically on CI when the output that is supposed to trigger the test's completion never arrives at the sink.
Such an example is available at https://circleci.com/gh/WallarooLabs/wallaroo/4846.

The test is failing because not all expected output arrives at the sink.

Why not all output arrives at the sink is unknown at this point, and should be considered a high priority bug.

Unfortunately, I don't know of a reliable way to reproduce this bug quickly.

[SeanTAllen]

@rachelreese ping. so this is on your radar.

[nisanharamati]

This has come up in #2297. Andy and myself are investigating, but so far no luck isolating the cause.

[nisanharamati]

This turned out to be a bug in ponyc, which is addressed in ponylang/ponyc#2840 by @dipinhora.
I'll close this issue once the bugfix is merged and our CI is updated to the new ponyc release.