[Feature]: Container images are deleted alongside packages

[gcapizzi]

Background

As an operator
I want to only store container images for packages currently used by the system
So that I don't face increasing costs of storage for running the same amount of apps

Acceptance Criteria

GIVEN I have a package
AND I have uploaded sources for it
WHEN I delete the package
THEN I see the corresponding container image containing the source has been deleted

Dev Notes

See #2112 (comment) for details.

[kieron-dev]

We are implementing this by adding a finalizer to CFPackage that deletes the container image from the registry using the docker registry v2 API using the go-containerregistry library. Some interesting observations:

• Google Artifact Registry will error when deleting an image manifest by digest when there are tags referencing that digest. For that reason we first attempt deleting all tags referencing the digest.
• ECR deletes the corresponding manifest when the tag is deleted (if there are not other tags referencing it). So with ECR we see errors when we attempt to delete the manifest as it has already gone. So we should see if we can handle a not found error differently.
• GCR and ACR effectively delete the repository when it contains no images. (I suspect they don't really have a concept of repository and just dynamically show folders in their UIs.)
• DockerHub does not allow deletion of image manifests via the docker registry v2 API - you have to use its proprietary API 😦