composefs?? random ENOENTS in CI

[edsantiago]

Seeing a lot of these in my no-flake-retry PR, only on rawhide root, only today:

something something
Error: copying layers and metadata for container "cid":  \
    initializing source containers-storage:test1: \
    extracting layer "sha": \
    lgetxattr /tmp/CI_RQPB/podman-e2e-827543605/imagecachedir/overlay/sha/merged/usr/bin/killall: no such file or directory

or

Error: exporting root file-system diff for "sha": \
    lgetxattr /tmp/CI_RQPB/podman-e2e-827543605/imagecachedir/overlay/sha/merged/usr/bin/bunzip2: no such file or directory

or llistxattr or even just open. There's also a very weird one in podman diff that does not match the same pattern but I would bet is related.

Today's changes are (1) local registry and (2) test composefs on rawhide.

• rawhide : int podman rawhide root host sqlite
  • 07-11 09:14 in Podman commit podman commit with volumes mounts and no include-volumes
  • 07-11 09:14 in Podman pod create podman create pod with --infra-image
  • 07-11 09:14 in Podman checkpoint podman restore multiple containers from single checkpoint image
• rawhide : int remote rawhide root host sqlite [remote]
  • 07-11 09:14 in Podman run entrypoint podman run user entrypoint with command overrides image entrypoint and image cmd

x	x	x	x	x	x
int(4)	podman(3)	rawhide(4)	root(4)	host(4)	sqlite(4)
	remote(1)

[edsantiago]

One more run of my PR, lots more failures. This is looking really bad for composefs.

Note that one of those failures is in diff and it exhibits the same ENOENT symptom.

• rawhide : int podman rawhide root host sqlite
  • 07-11 13:45 in Podman checkpoint podman checkpoint and restore container with root file-system changes
  • 07-11 13:45 in Podman run podman run --seccomp-policy image (bogus profile)
  • 07-11 13:45 in Podman build podman build --from, --add-host, --cap-drop, --cap-add
  • 07-11 09:14 in Podman commit podman commit with volumes mounts and no include-volumes
  • 07-11 09:14 in Podman pod create podman create pod with --infra-image
  • 07-11 09:14 in Podman checkpoint podman restore multiple containers from single checkpoint image
• rawhide : int remote rawhide root host sqlite [remote]
  • 07-11 13:46 in Podman diff podman image diff
  • 07-11 09:14 in Podman run entrypoint podman run user entrypoint with command overrides image entrypoint and image cmd

x	x	x	x	x	x
int(8)	podman(6)	rawhide(8)	root(8)	host(8)	sqlite(8)
	remote(2)

[edsantiago]

Another CI run, another bunch of failures, I'm not going to bother posting them all.

One curious finding: so far, no failure in sys tests. Could it be that the magic command-line --pull-option is not working? Maybe one of those options is necessary, but e2e tests aren't actually getting them?

[Luap99]

I don't think the problem would be the cli option. I think one major difference is that the e2e tests use this special imagestore setup on the the main regular store for images to share images

[Luap99]

@giuseppe PTAL

[giuseppe]

sorry for not catching this at review time, we need to pass each flag with a different --pull-option: containers/podman#23257, that is not a fix for this case though, since we want to support non-composefs additional stores as well.

I've also bumped the c/storage dependency as there are two fixes for composefs that might be helpful in this case.

I am still investigating the problem though, trying to reproduce locally.

[edsantiago]

I've rebased my no-retries PR, on main which now includes containers/podman#23257, and I'm sorry to say that the flake persists.

[giuseppe]

potential fix: #2036, I am still validating it

I'll update here when it is ready for testing/review