Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

image-layout: move refs to refs/tags #219

Closed
wants to merge 1 commit into from

Conversation

philips
Copy link
Contributor

@philips philips commented Aug 29, 2016

Based on the confusion from Antonio in the image-layout in
#173 we are going to
move everything that was in refs to refs/tags. This does a few
things:

  1. It makes it clearer that this layout is for a single "image name" instead of allowing for lots of different refs.
  2. It matches the layout of git so it likely matches existing expectations.

This was also discussed during the OCI F2F and decided to be a
reasonable idea.

Signed-off-by: Brandon Philips brandon.philips@coreos.com

Based on the confusion from Antonio in the image-layout in
opencontainers#173 we are going to
move everything that was in `refs` to `refs/tags`. This does a few
things:

1. It makes it clearer that this layout is for a single "image name" instead of allowing for lots of different refs.
2. It matches the layout of git so it likely matches existing expectations.

This was also discussed during the OCI F2F and decided to be a
reasonable idea.

Signed-off-by: Brandon Philips <brandon.philips@coreos.com>
@philips
Copy link
Contributor Author

philips commented Aug 29, 2016

@philips philips added this to the v0.5.0 milestone Aug 29, 2016
@runcom
Copy link
Member

runcom commented Aug 29, 2016

Great! I like refs/tags also, thx @philips. LGTM (IANAM)

@wking
Copy link
Contributor

wking commented Aug 29, 2016

On Mon, Aug 29, 2016 at 09:44:58AM -0700, Brandon Philips wrote:

Based on the confusion from Antonio in the image-layout in
#173 we are going to
move everything that was in refs to refs/tags. This does a few
things:

  1. It makes it clearer that this layout is for a single "image name"
    instead of allowing for lots of different refs.

Previous discussion of this around 1. And the current spec has (and
I like) 3:

No semantic restriction is given for object names in the refs
subdirectory.

I see no reason to back away from that at the image-layout level. And
I don't see why there would be a problem with folks keeping all
their refs in a single image-layout (although they probably wouldn't
want to back it with tar ;). For example:

https://oci.example.com/refs/…
https://oci.example.com/blobs/…

could have multiple debian:… refs, multiple ubuntu:… refs, multiple
python:… refs, etc.

Folks with special per-blob auth rules might choose to use per-"image
name" image-layout instances, but and there's no reason you couldn't
do that now. But I think we want to keep support for multi-"image
name" image-layout instances too.

  1. It matches the layout of git so it likely matches existing
    expectations.

Git also has refs/heads and refs/remotes. Do we expect image-layout
to need those, or other kinds of refs? I don't see a use case for
them, excepting possibly a refs/HEAD working like Git's .git/HEAD to
provide a default entry point for folks running an image with multiple
refs but not specifying a ref directly.

 Subject: Misleading language in image-layout.md

@sgotti
Copy link

sgotti commented Aug 29, 2016

I was also thinking that the oci image layout could also work for multiple images and the ref is not tied to a name (or just a tag) but it was a level below image name. For example you can just create a simple image "registry" using an oci image layout using a filesystem backed by http, nfs, local fs access etc...
Instead retrieving which location and ref refers to example.com/app:tag will be an upper layer that does image discovery.

I also tried to explain some possibilities of this approach in the Distribution point concept for rkt (rkt/rkt#2953) where there are two different distribution types: a direct distribution point called oci-image-layout and one or more theoretical (not currently covered by the image spec) indirect (that will do location and ref discovery starting from an image name) oci distributions.

@vbatts
Copy link
Member

vbatts commented Aug 29, 2016

LGTM

Approved with PullApprove

@philips
Copy link
Contributor Author

philips commented Aug 29, 2016

@wking the problem is that we don't allow slashes or colons in ref names to ensure we don't confuse filesystems. Without those separators it is strange to imagine inventing some sort of serialization from the common debian/base:v0.3.0 pattern or gcr.io/foo/bar:v1.0 into the refs directory.

@wking
Copy link
Contributor

wking commented Aug 29, 2016

On Mon, Aug 29, 2016 at 01:52:07PM -0700, Brandon Philips wrote:

@wking the problem is that we don't allow slashes or colons in ref
names to ensure we don't confuse filesystems. Without those
separators it is strange to imagine inventing some sort of
serialization from the common debian/base:v0.3.0 pattern or
gcr.io/foo/bar:v1.0 into the refs directory.

There's a straightforward fix for that in #174. Even without that
fix, there are still reasonably straightforward ways to put multiple
names in one image-layout (moby/moby#25779).

But regardless, “we're not sure how to name refs when there are
multiple image names in one image-layout” doesn't seem like it
directly motivates “let's add a tags/ directory”. It sounds more like
motivation for “let's add semantic restrictions to refs”, and I don't
think image-layout is the place to add those restrictions. I'm fine
with a higher-level policy section in image-spec that adds these
restrictions.

Folks who can find an ref naming scheme that they're comfortable with
([1], #174, moby/moby#25779) should be allowed to pack several
image-names into a single image-layout without breaking image-layout
compliance (although they may break image-spec compliance).

@@ -1,13 +1,13 @@
## Open Container Initiative Image Layout Specification

The OCI Image Layout is a slash separated layout of OCI content-addressable blobs and [location-addressable](https://en.wikipedia.org/wiki/Content-addressable_storage#Content-addressed_vs._location-addressed) references (refs).
The OCI Image Layout is a slash separated layout of OCI content-addressable blobs and [location-addressable](https://en.wikipedia.org/wiki/Content-addressable_storage#Content-addressed_vs._location-addressed) reference tags (tags).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean we are dropping the concept of generic refs? I thought we were just calling out a "Reserved" structure within refs?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Mon, Aug 29, 2016 at 02:07:10PM -0700, Stephen Day wrote:

@@ -1,13 +1,13 @@

Open Container Initiative Image Layout Specification

-The OCI Image Layout is a slash separated layout of OCI content-addressable blobs and location-addressable references (refs).
+The OCI Image Layout is a slash separated layout of OCI content-addressable blobs and location-addressable reference tags (tags).

Does this mean we are dropping the concept of generic refs? I
thought we were just calling out a "Reserved" structure within
refs?

Ugh. In order of my decreasing preference:

a. Keep our current generic refs 1.
b. Replace generic refs with refs that have some required tag
semantics (this PR?).
c. Support both generic refs and refs with required tag semantics
(what you seem to be suggesting with the “Reserved structure”
comment).

Any type information about refs should go in the descriptor media
type field. I see no reason to bake it into the bucket that holds the
descriptors.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the use case of "generic refs"? I thought we were just trying to clear up the use case of the refs directory by making the tags use case explicit through naming, as git does.

The alternative proposal we discussed was just renaming "refs/" to "tags/".

Copy link
Contributor

@wking wking Aug 29, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Mon, Aug 29, 2016 at 02:24:21PM -0700, Brandon Philips wrote:

What is the use case of "generic refs"?

It allows image-layout users to link link other content if they see a
use case for it (vs. requiring refs to point to a manifest(-list) with
some standard name mapping). For example, Git has a tagged blob 1,
because that was a convenient way to distribute Junio's public key.

Having multiple image-names in one image-layout is a similar use-case
that I think we want to allow at the image-layout level.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is calling this a "tag" stopping you from having multiple image-names? I view this as similar to how GitHub has the gh-pages branch. We aren't restricting the use cases but trying to clearup the intended use case.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Mon, Aug 29, 2016 at 03:02:46PM -0700, Brandon Philips wrote:

I don't know. I simply don't understand why having a directory
called refs/tags is confusing.

You'll have people talking about refs and people talking about tags
and people wondering if there's some distinction between them (which
it sounds like there isn't).

What is your use case for having lots of refs pointing to different
images in a single image-layout?

I'm publishing a bunch of images on example.com, and don't want to be
bothered with maintaining multiple image-layout instances 1. Or I
want a convenient way to distribute my public key along with the
images it signs 2. And supporting generic refs protects
image-layout implementations from having to worry about scope creep
and such if a more opinionated refs spec decided that it needed to
support something different from the narrow use case it initially
picked.

rkt image import --name mystuff.com/foo/bar /path/to/oci-image-layout
docker import --name mystuff.com/foo/bar /path/to/oci-image-layout

In these cases the name and tags make sense. What is your use case?

I'd rather keep both explicit:

$ my-tool import LAYOUT-PATH REF [NAME]

unless we decide to support something like HEAD 1, in which case REF
would also be optional. If name was not specified on the command
line, the name used for the imported object would come from some
verifiable name (e.g. using the Merkle objects discussed in #176, see
3).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Mon, Aug 29, 2016 at 3:16 PM W. Trevor King notifications@github.com
wrote:

I'd rather keep both explicit: $ my-tool import LAYOUT-PATH REF [NAME]
unless we decide to support something like HEAD [1], in which case REF
would also be optional. If name was not specified on the command line, the
name used for the imported object would come from some verifiable name
(e.g. using the Merkle objects discussed in #176
#176, see [3]). [1]: #219
(comment)
#219 (comment)
[2]: #219 (comment)
#219 (comment)
[3]: #173 (comment)
#173 (comment)

Eh, I think that it is going to be more common for someone to download "an
entire image named foo" then wanting to import the whole thing and expect
the tags to show back up rather than forcing the user to specify every ref.
For example you would actually need to do:

my-tool import /path/to/layout my-img-foo-bar-v1.0.0 my.img/foo/bar v1.0.0
my-tool run my.img/foo/bar:v1.0.0

vs

my-tool import /path/to/layout my.img/foo/bar
my-tool run my.img/foo/bar:v1.0.0

I feel there is a usability cost to supporting lots of imgs in a single
layout on this common import UX.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Mon, Aug 29, 2016 at 03:24:32PM -0700, Brandon Philips wrote:

Eh, I think that it is going to be more common for someone to download "an
entire image named foo" then wanting to import the whole thing and expect
the tags to show back up rather than forcing the user to specify every ref.
For example you would actually need to do:

my-tool import /path/to/layout my-img-foo-bar-v1.0.0 my.img/foo/bar v1.0.0
my-tool run my.img/foo/bar:v1.0.0

vs

my-tool import /path/to/layout my.img/foo/bar
my-tool run my.img/foo/bar:v1.0.0

I feel there is a usability cost to supporting lots of imgs in a single
layout on this common import UX.

So add an:

$ my-tool import-all LAYOUT-PATH

that imports all the refs (still getting names from the CAS blobs
1). Or, without a HEAD spec, make:

$ my-tool import LAYOUT-PATH

(without an explicit REF) import all the refs.

Specifying an unverified name on the command line or extracting
information from the ref name (vs. the referenced CAS content)
should not be part of your main UX.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How would you get a name from a CAS blob?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Mon, Aug 29, 2016 at 04:19:33PM -0700, Brandon Philips wrote:

How would you get a name from a CAS blob?

See #176 and 1.

@philips
Copy link
Contributor Author

philips commented Aug 29, 2016

I was perfectly fine with the status quo of having refs be flat with no implied use case. If we can just add some clarifying language about them mapping to tags in a general case for import/export from docker engines would that be sufficient @runcom ?

@runcom
Copy link
Member

runcom commented Aug 29, 2016

@philips, yes that was my plan at the beginning (just a language issue)

@philips
Copy link
Contributor Author

philips commented Aug 31, 2016

I am going to close this out and open something that tries to clear up the language issue instead.

@philips philips closed this Aug 31, 2016
@vbatts
Copy link
Member

vbatts commented Sep 15, 2016

@philips was a new issue opened for this yet?

@wking
Copy link
Contributor

wking commented Sep 15, 2016

On Thu, Sep 15, 2016 at 07:00:39AM -0700, Vincent Batts wrote:

@philips was a new issue opened for this yet?

#225, since merged.

@vbatts
Copy link
Member

vbatts commented Sep 15, 2016

nah. the use of ./refs/tags/foo

@wking
Copy link
Contributor

wking commented Sep 15, 2016

On Thu, Sep 15, 2016 at 07:30:55AM -0700, Vincent Batts wrote:

nah. the use of ./refs/tags/foo

Why would you do that? Can't higher levels pick a convention like
tag-foo just like 1 if they're worried about mixing different sorts
of refs?

@vbatts
Copy link
Member

vbatts commented Sep 15, 2016

like has been mentioned before, for the same reason git does it. a tag is just one reference type. there could be others. this is the same thing i said yesterday on the call.

@wking
Copy link
Contributor

wking commented Sep 15, 2016

On Thu, Sep 15, 2016 at 07:49:49AM -0700, Vincent Batts wrote:

like has been mentioned before, for the same reason git does it. a
tag is just one reference type. there could be others.

And higher levels can namespace as many as they like using the
existing set of allowed characters. Why special case a slash for this
one class of refs?

@vbatts
Copy link
Member

vbatts commented Sep 15, 2016

sometimes i'm not even sure the point you're trying to make. Honestly.

@wking
Copy link
Contributor

wking commented Sep 15, 2016

On Thu, Sep 15, 2016 at 07:57:28AM -0700, Vincent Batts wrote:

sometimes i'm not even sure the point you're trying to make. Honestly.

We currently have one class for all refs. Higher levels which want to
divide that into multiple classs can already namespace however they
wish as suggested here 1. That approach works for “I want a
subclass for just base-64-encoded refs” (you prefix with
‘org.prefix.base64-’ or ‘base64-’ or whatever). It also works for “I
want a subclass for just tags” (you prefix with ‘org.prefix.tag-’ or
‘tag-’ or whatever). This is the same approach taken by RFC 5890 with
A labels (prefixed with ‘xn--’) 2.

If you instead bless a subclass with a slashed name (like ‘tags/’)
associated tooling and APIs need to grow to handle the additional
class. That seems like more complication then it's worth, when you
could accomplish the same goal by defining a ‘tag-’ prefix and make it
easy for tooling to handle your refs as opaque bytes without having to
grow special slash handling.

So I'm suggesting we either leave the spec as it is on this point or
define a ‘tag-’ prefix, but that we don't define a ‘tag/’ prefix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants