CI flake: podman run --conmon-pidfile: conmon not running?

[edsantiago]

One of the system tests does podman run --conmon-pidfile=PATH then reads pid from PATH and confirms that /proc/pid/exe is a symlink pointing to .*/conmon. This test has flaked twice in the past month, with:

#/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
#|     FAIL: conmon pidfile (= PID 100491) points to conmon process
#| expected: '.*/conmon'
#|   actual: ''
#\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The two flakes:

• fedora-32 : special_testing_rootless
  • PR Bump github.com/containers/storage from 1.23.2 to 1.23.4 #7539
    • 09-05 07:13
• fedora-32 : special_testing_rootless [remote]
  • PR Don't limit the size on /run for systemd based containers #7346
    • 08-18 17:42

Source code:

podman/test/system/030-run.bats

Line 178 in a82de0e

is "$(readlink /proc/$conmon_pid/exe)" ".*/conmon" \

This is so infrequent that it's hard to know where to start. I'm filing because:

• Someone might perhaps think of a race condition in the conmon-pidfile code (e.g. perhaps the code is not flushing/closing the output file, so my test is reading an incomplete PID? Maybe the PID above should've been 100491X?)
• Someone could suggest a way for me to instrument the test, to make the next failure easier to diagnose?

[rhatdan]

This looks like the old race condition where we were not getting stdout back on the remote end. I believe that is fixed?

[edsantiago]

You mean #7195? That seems doubtful because one of the failures is non-remote, and regardless, the missing output is from readlink which is not running under podman. Did I misunderstand?

[rhatdan]

Ok, I did not read the actual code, I just saw the failure and it looked familiar.

[rhatdan]

Would this happen if $conmon_pid nolonger existed?

[edsantiago]

Yes. Is that possible in a sleep infinity container?

[rhatdan]

One would hope not, but that is the symptom

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@edsantiago is this still an issue? I see a couple of mergers that reference this issue?

[edsantiago]

@giuseppe explicitly mentions in containers/crun#505 that it does not address this issue.

But I have not seen it since Sept 10, when I added two statements (an echo and an ls) intended to help debug the flake. My hunch is that the flake is a race condition, very likely caused by trying to read the pidfile before conmon has written it, and that my debug statements take just long enough to mask this. Absent an explicit diagnosis and fix, I'm not willing to close this, because that just seems to me like sweeping it under the rug. I will not interfere with you or someone else closing it, though. If it bites someone in the community, I trust they will comment here and someone can reopen.

[rhatdan]

I will close and we can reopen when it happens again.