Support the removal of images

[mssola]

Work is being done upstream here. Therefore, this feature, although needed, is blocked by it. We will work on it once that issue is fixed. It's supposed to get out with distribution 2.1

Update: distribution 2.1 has been released and soft deletes have made it in the release. Here's a checklist in order to fix this issue:

• Implement soft deletes in the registry client.
• Implement deletes in the UI.

[mssola]

Soft deletes support has been implemented in the registry client with this commit e2de2ec. However, we still need to investigate a bit further how it works under the hood before implementing it on the UI.

[JonathonReinhart]

For reference, the starting point for the "Distribution supporting deletes" discussion seems to be here: https://github.com/docker/distribution/blob/master/ROADMAP.md#deletes

[hhcordero]

Not sure if this is the correct place to ask, is there any update on this feature? In our company, we are evaluating Portus and it seems that this missing feature is really important for maintaining the registry.

[mssola]

@hhcordero sadly, this is not implemented in docker distribution (the registry) yet. That being said, I'd say that there is work being done in this regard (e.g. pruning). So, hopefully in the near future we will be able to somehow tackle this.

[JonathonReinhart]

It looks like Portus could probably take advantage of the soft-deletes implemented in distribution/distribution#461 now, and let the registry deal with the cleanup under the hood (whenever they get around to it). But note that the reference argument must be a digest and not a tag (See distribution/distribution#549). I've asked for clarification but haven't received any.

[mssola]

@JonathonReinhart note that soft deletes are already implemented in Portus itself (take a look at the bin/client.rb file).

and let the registry deal with the cleanup under the hood (whenever they get around to it)

That's the deal for me. We could implement deletes, but that would be lying until there is no pruning or GC support on registry's side. Thus, I prefer to wait for that.

[JonathonReinhart]

Valid point. The one benefit would be removal of the tags, minimizing visual "clutter".

[cyphar]

@JonathonReinhart Yeah, but then you'll run into the problem of "I've deleted all of these tags! Why has my sever run out of disk space?". IMO, the principle of least surprise requires us to not implement a feature that actively lies to users, at least until distribution get around to auto-GCing orphaned tag references.

[JonathonReinhart]

These points all sound perfectly valid to me. Maybe all of these issue cross-references will get someone's attention over at Docker :-)

[cyphar]

One can only hope. I could prod some of the maintainers. 😸

[pjdarton]

I'm in complete agreement w.r.t. "principle of least surprise" - if the backing store is a cluttered mess, it should look like the cluttered mess it is. One needs to know where the disk space has gone.

FYI the reason I found Portus was because I was searching for a (stable) Docker Registry that could handle deletions (v1 registry isn't stable, v2 isn't complete). Lack of support for deletion is a killer issue for me - contrary to popular belief, disk space is not infinite. My CI builds that create docker images can churn through terabytes a week.
If Portus provided a docker repository API and the ability to delete images, I would have adopted it.

@cyphar If you can prod docker folks to raise their awareness of this issue, please do so...

[Seb-Solon]

@pjdarton I have the exact same issue.
My current workaround for now using this script as base : https://github.com/burnettk/delete-docker-registry-image and wrap it with some specific stuff to remove my tag/image in a crontab
Hope it helps.

Regarding this issue : +1 for waiting the feature to be fully working on docker side.

[JonathonReinhart]

I could prod some of the maintainers. 😸

,

@cyphar If you can prod docker folks to raise their awareness of this issue, please do so...

Not to speak on their behalf, but the Docker maintainers are very, very aware of this issue, and I'm sure they are getting tired of the continuous stream of +1's being thrown at every issue that talks discusses image deletion. See this comment as an example: docker-archive/docker-registry#45 (comment)

I would definitely suggest having a look at the deletes section of the roadmap before attempting to engage with them about this issue, otherwise you'll quickly be dismissed.

That being said, I find it quite frustrating that a project as large and corporate-sponsored as Docker hasn't figured out a way to delete things from the registry yet. I mean, would the EXT4 developers ever release a version of their filesystem that didn't let you delete files from it?! So by all means, if SUSE personnel would be able to express that Portus remains incomplete because of Docker shortcomings, please do so.

[pjdarton]

I'm aware that it's a non-trivial issue, and have some sympathy for the developers who are at the sharp end of this criticism, but I share your frustration that it got released in this incomplete state.
It's advertised as an implementation of the Docker v2 registry, but it's missing mandatory elements of that API. That was quite a surprise...

If Portus is just bundling the official Docker v2 registry implementation then Portus cannot claim to be a Docker v2 registry until the deletion part of the API is implemented (either by Portus or within the official Docker v2 registry code).

[mssola]

If Portus is just bundling the official Docker v2 registry implementation then Portus cannot claim to be a Docker v2 registry until the deletion part of the API is implemented (either by Portus or within the official Docker v2 registry code).

@pjdarton We do not claim to be a Docker v2 registry. Portus is an authorization service for a Docker Registry implementing v2 (a.k.a. docker distribution). The description of Portus is:

Authorization service and frontend for Docker registry (v2)

More specifically, in the first image of this document, Portus fills the "Authorization Service" box, nothing more, nothing less. Therefore, Portus can only support deletion when docker distribution does it ;)

I would definitely suggest having a look at the deletes section of the roadmap before attempting to engage with them about this issue, otherwise you'll quickly be dismissed.

I agree with @JonathonReinhart on this.

[mssola]

GC support has just been merged in docker distribution and it will be there for 2.4 (see PR). I will investigate on it, and we will hopefully get rid of this issue.

[rennhak]

As a temporary work around this worked for us.
https://github.com/burnettk/delete-docker-registry-image

[mssola]

After doing some work on top of distribution 2.4, I've decided that this is in fact no longer blocked. I've already sent a couple of PRs in preparation of the "big" feature :)

[holgerreif]

Since it's such a big feature (as you pointed out by yourself ;-) wouldn't it be worth to issue a interim 2.0.4 release containing especially this feature? Waiting until autumn is quite a long time...

[mssola]

@holgerreif I'd say no for two reasons:

1. We've already done some previous work before working on this, so separating the commits related to this feature into its own branch/tag would be troublesome (possibly adding regressions with the porting 😅)
2. We reserve patch-level releases for bug fixes only. This would be against our release procedure.

So, I can only see two options: have it on master until 2.1 is released (the usual procedure); or having a separate branch with the current master plus this feature. In this second option master would still have this feature, so the only gain would be to have a branch with the least amount of other features in it plus this one. This is maybe the best option. What do you think ?

[holgerreif]

Then I'll stick with master.

@JonathonReinhart @rennhak @pjdarton @Seb-Solon does anybody of you have opinions?

[JonathonReinhart]

Since it's such a big feature... wouldn't it be worth to issue a interim 2.0.4 release containing especially this feature?

@holgerreif You've contradicted your self here, by suggesting a "big feature" to into a "patch" release. Have a look at Semantic Versioning, which describes what types of things go into what type of release.

Personally, I'm on 2.0, but plan to move to master to get some other recent fixes. Of course, I wish this project were moving faster and 2.1 were already out, but the only way to hurry open-source is with your time or money, and I don't have a lot of either at the moment 😊

[mssola]

@holgerreif I've been doing some work around this feature this week and the previous one. I expect to get the feature finished for either this week or the next one. So hopefully you'll be able to test this feature soon enough :)

[mssola]

It's now possible to remove images & tags from the Portus UI. In the following weeks I'll add support for removing namespaces, teams and users, and I'll try to polish any corner case that might exist. Documentation for this is still to be written, but a summary would be:

1. You have to enable deletes on the Portus config (which should only be activated if you have distribution 2.4, or if you don't care about disk space 😅)
2. Users with the required privileges will be able to remove images/tags.
3. Administrators are still required to run GC manually once in a while on the registry. The reason for this is that GC is not available from the registry's API, so there's no way from Portus to perform this.
4. You don't have to enable maintenance mode in the registry to use GC, but it's discouraged because it may end up with inconsistencies due to race conditions (we are working on providing a workaround for this).

[EugenMayer]

@mssola could you at least tell me what configuration to add so i can use this feature?

[mssola]

@EugenMayer certainly. For clarification for other users as well, in my previous comment, when I say:

1. You have to enable deletes on the Portus config (which should only be activated if you have distribution 2.4, or if you don't care about disk space 😅)

I mean that we introduced the delete configurable value. It only has the enabled key, which is set to false by default. On master, you can see it here.

[EugenMayer]

@mssola if found it already, thank you. My issue was, that i was using https://github.com/sshipway/Portus which is rather a fork then just a build for https://hub.docker.com/r/sshipway/portus/ .. and it was very outdated and did not include this feature at all

i went forward and made a clean build repo, which uses SUSE/portus without modifications .. publishing them to https://hub.docker.com/r/eugenmayer/portus/ while building is done using https://github.com/EugenMayer/portus-build

If that is out of any interest for you - rip it off :)

I already tried that feature and it works great. If you wonder what i am doing, i am packing portus for rancher.io so it is installable as a "app" ( catalog ) https://github.com/EugenMayer/kontextwork-catalog/blob/master/templates/registry-slim/1/docker-compose.yml

[mssola]

@mssola if found it already, thank you. My issue was, that i was using https://github.com/sshipway/Portus which is rather a fork then just a build for https://hub.docker.com/r/sshipway/portus/ .. and it was very outdated and did not include this feature at all

i went forward and made a clean build repo, which uses SUSE/portus without modifications .. publishing them to https://hub.docker.com/r/eugenmayer/portus/ while building is done using https://github.com/EugenMayer/portus-build

One of the efforts that we are doing for the 2.1 release is to come up with an official production-ready Docker image. For this, we've prepared this image, which has three tags:

• head: Users that want to be on the bleeding edge.
• 2.0: Users that want to be on the current stable release of 2.0.x.
• 2.0-git: Users that want to be on the current stable release of 2.0.x + some patches that are to be released.

We are still working on the documentation of this image, but an overview of it is already available here. I'd recommend giving this image a try.

Edit: proper URL for the documentation

[EugenMayer]

great! i could just create a wrapper-image to offer a different entry-point ( due to the configuration over ENV ), but in general, i would be very glad to use your official image!

[EugenMayer]

Just had a look at the image and created some feedback #912 for now, that image would be not be practical to build on for this purpose. Maybe we can work on this