Reenable remote system tests

[edsantiago]

podman-remote is in better shape now. Let's see what needs
to be done to reenable remote system tests.

Signed-off-by: Ed Santiago santiago@redhat.com

[edsantiago]

This is uncomfortable: the remote test timed out but with no actual log of how far it got:

+ make remotesystem
# Wipe existing config, database, and cache: start with clean slate.
rm -f -rf /home/some13225dude/.local/share/containers /home/some13225dude/.config/containers
# Start podman server using tmp socket; loop-wait for it;
# test podman-remote; kill server, clean up tmp socket file.
# podman server spews copious unhelp
Timed out!

Restarted in hopes that it's a purely-coincidental infrastructure flake.

[edsantiago]

@cevich I need help here please. The failures look like real timeouts, but the logs are useless (incomplete). I have to assume there's a buffering problem, as if stdout/stderr aren't getting flushed properly. Does that sound familiar? Can you think of any way we can get full logs?

[cevich]

@edsantiago taking a peek...

[cevich]

@edsantiago this is a tricky one! The main thing that stands out to me here is the size of the onion. Maybe try peeling back a layers? For example, the wait/retry loop in a makefile is two complexities wrapped in an enigma. I know make has it's fingers into the stdio and exit-code pie, what happens w/o make being involved?

[edsantiago]

I think the Ubuntu issue is solved: Ubuntu /bin/sh is not bash, so [[ and &> weren't working. Those are resolved, and Ubuntu 19 & 20 tests now passed.

I see two alarming failures in special_testing_rootless:

[+0257s] not ok 74 podman volume with --userns=keep-id
[+0257s] # $ /var/tmp/go/src/github.com/containers/podman/bin/podman-remote --url unix:/tmp/podman.fpYZ0P run --rm -v /tmp/podman_bats.LsZK3v/volume_O8zRoMsGmt:/vol:z quay.io/libpod/alpine_labels:latest stat -c %u:%s /vol/myfile
[+0257s] # read unixpacket @->/run/user/23298/libpod/tmp/socket/1bee53f8e19e2b03ff773e504fead7f5994b8be5545b315a73a3d2cef290f567/attach: read: connection reset by peer
[+0257s] # #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
[+0257s] # #|     FAIL: w/o keep-id: stat(file in container) == root
[+0257s] # #| expected: '0:0'
[+0257s] # #|   actual: 'read unixpacket @->/run/user/23298/libpod/tmp/socket/1bee53f8e19e2b03ff773e504fead7f5994b8be5545b315a73a3d2cef290f567/attach: read: connection reset by peer'
[+0257s] # #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

-------------------------------------

[+0297s] not ok 88 sensitive mount points are masked without --privileged
[+0297s] # $ /var/tmp/go/src/github.com/containers/podman/bin/podman-remote --url unix:/tmp/podman.fpYZ0P run --rm quay.io/libpod/alpine_labels:latest stat -c%n:%F:%h:%T:%t /dev/null /proc/acpi /proc/kcore /proc/keys /proc/latency_stats /proc/timer_list /proc/sched_debug /proc/scsi /sys/firmware /sys/fs/selinux /sys/dev
[+0297s] # can only attach to created or running containers: container state improper
[+0297s] # #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
[+0297s] # #| FAIL: can only attach to created or running containers: Unknown file type ' container state improper'
[+0297s] # #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The first: where is that /run/user/.../socket coming from??? The podman command is very clearly invoked with --url. What is going on here?

The second, well, I'm just going to disable that test for podman-remote - but what is happening here too? It's just a podman run, no attach anywhere. I think this one is a race condition, but I'm not going to dive into it now. I want to document these now and hope we don't ignore them.

[edsantiago]

Okay, I don't think podman-remote is ready for prime time. These failures are weird. Giving up for the week.

[cevich]

Ubuntu /bin/sh is not bash

I can't tell you the number of times this catches me...a genuine PITA for no good reason IMHO...or at least no good reason I'm aware of. I'm glad to read that you got past the odd hanging/garbage problem.

The podman command is very clearly invoked with --url. What is going on here?

Also odd, I wonder if we're running into our old IFS bats bug again? IIRC, that was causing command-line args to not get passed through properly. In any case, it should be possible reasonably easy to run the command manually and see if it reproduces, no?

I think this one is a race condition

That occurred to me as well, like perhaps it's exiting just at the wrong time. Could it be due to a hiccup talking to quay.io maybe? Otherwise, given it's proximity to the first failure, I wonder if the two are somehow related. Is it possible the remote-end got wedged or crashed in some way causing both failures?

[edsantiago]

Just a marker to list the issues found and filed (so far) by this PR: #7116 #7117 #7119 #7122 #7123 #7128 #7135 #7136 #7137 #7139 #7195

[edsantiago]

This is ready for review - or at least as close as I can get it. Unfortunately, I just don't think podman-remote is ready for the world: it's not reliable. This PR has seen a history of flakes, most particularly what I believe is a race condition in which the client never sees output (#7195). Until that is resolved, I think it's a bad idea to enable podman-remote tests in CI: we will get a ton of flakes. But I also think it's a bad idea to leave podman-remote untested. Comments welcome.

[vrothberg]

LGTM
@baude @jwhonce @mheon PTAL

[vrothberg]

Thanks, @edsantiago! Your findings are concerning and suggest to dedicate some extra attention to podman-remote soon.

[rhatdan]

/approve
@edsantiago I don't really understand the comments, are you saying you don't want this PR merged?

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: edsantiago, rhatdan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rhatdan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[edsantiago]

@rhatdan I'm ambivalent. If we merge it, there is going to be a lot of developer pain because of frequent flakes and reruns. But I also don't see any incentive to fix podman-remote, so if we don't merge it I can see us going another six months without testing and with more bugs creeping into podman-remote.

I lean toward merging, I just want everyone to know that it's going to be painful if we do. (Until the flake bug is fixed).

[rhatdan]

/lgtm