podman image exists, with vfs.imagestore and image from quay, fails

[edsantiago]

I have no idea how to even describe this, so someone please feel free to change the title to a more descriptive one.

Discovered during testing of #8165 -- a podman setup using vfs and vfs.imagestore will, under certain circumstances involving images from quay.io, falsely claim false on image exists. A reproducer is worth a thousand of my words:

#!/bin/bash

set -ex

PROOT=/dev/shm/mypodman
rm -rf $PROOT
mkdir -p $PROOT

# What's really weird is that this test *succeeds* with docker.io
IMAGE=quay.io/libpod/alpine:latest
#IMAGE=docker.io/library/alpine:latest

# Pull something in order to initialize imagestore
# Without this initial pull, the last pull below fails with:
# Error: error opening "/dev/shm/mypodman/cache/vfs-images/images.lock": no such file or directory
PODMAN="./bin/podman --root $PROOT/cache --storage-driver vfs"
$PODMAN pull -q $IMAGE

# And now, switch --root to something empty, and set imagestore to the above.
# Pull the same image, and boom.
# If you comment out the 'pull' below, 'image exists' succeeds.
PODMAN="./bin/podman --storage-opt vfs.imagestore=$PROOT/cache --root $PROOT/root --storage-driver vfs"
$PODMAN pull -q $IMAGE

# Shows two images, differing only in the rightmost 'R/O' column
$PODMAN images -a

# This fails!
$PODMAN image exists alpine

[rhatdan]

/assign

[edsantiago]

I don't think this is fixed; the original test continues to fail. So does the reproducer in comment 0.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

This test is being screwed up by the mirroring.

If you run images, you see that you have these images.
REPOSITORY TAG IMAGE ID CREATED SIZE R/O
quay.io/libpod/alpine latest 961769676411 15 months ago 5.85 MB false
quay.io/libpod/alpine latest 961769676411 15 months ago 5.85 MB true

But the image exists is looking for
docker.io/library/alpine

Since the image exists does not go through the mirror code, it fails.

[rhatdan]

@vrothberg @mtrmac This is happening in containers/image GetStoreImage() call.

Should this be translating the local users name to the mirror name?

[rhatdan]

		img, err = is.Transport.GetStoreImage(ir.store, ref)

[mtrmac]

@rhatdan I assume by “mirror” you consistently mean vfs.imagestore, don’t you? There is no “mirror name” AFAICS, just two places to look up images with a name, and no way to name explicitly one or the other image if there is more than one in the multiple stores.

At a first glance, it’s not too surprising that docker.io/* and quay.io/* images behave differently (assuming quay.io is not in the unqualified-search-registries list), because the former one uses the

podman/libpod/image/image.go

Lines 462 to 482 in f01630a

	candidates, err := shortnames.ResolveLocally(sys, inputName)
	if err != nil {
	return "", nil, err
	}
	for _, candidate := range candidates {
	img, err := ir.store.Image(candidate.String())
	if err == nil {
	return candidate.String(), img, nil
	}
	}
	// Backwards compat: normalize to docker.io as some users may very well
	// rely on that.
	ref, err := is.Transport.ParseStoreReference(ir.store, inputName)
	if err == nil {
	img, err = is.Transport.GetStoreImage(ir.store, ref)
	if err == nil {
	return inputName, img, nil
	}
	}

path (including the hard-coded docker.io default at the bottom).

For quay.io, this triggers the undocumented findImageInRepotags heuristic (which, ceterum autem censeo , probably should not exist at all, but I can’t find any record of why it exists so I’m not sure).

---

Just reading the code, without testing it at all: AFAICS in the quay.io case:

• podman/libpod/image/image.go

  Line 486 in f01630a

  images, err := ir.GetImages()

  finds both images (same name, different backing stores)
• findImageInRepoTags finds both as matches, fails with errors.Wrapf(define.ErrMultipleImages, searchName)
• The caller in

  podman/libpod/image/image.go

  Lines 496 to 500 in f01630a

  	if err == nil {
  	return inputName, repoImage, nil
  	}
  	return "", nil, errors.Wrapf(ErrNoSuchImage, err.Error())

  just throws away that value and returns ErrNoSuchImage…
• … causing the error value check in

  podman/pkg/domain/infra/abi/images.go

  Lines 43 to 44 in f01630a

  	if errors.Cause(err) == define.ErrMultipleImages {
  	return &entities.BoolReport{Value: true}, nil

  to not trigger, and Exists to return false.