Misleading language in image-layout.md #173

runcom · 2016-07-21T13:44:18Z

Three issues:

First:

Object names in the refs and blobs subdirectories MUST NOT include characters outside of the set of "A" to "Z", "a" to "z", "0" to "9", the hyphen -, the dot ., and the underscore _.

This sentence is wrong because I can have colons : into the refs subdirectory to say ./refs/vbatts-app:latest

Second:

Hash algorithm identifiers containing the colon : will be converted to the hyphen -. For example sha256:5b will map to the layout blobs/sha256-5b.

these two sentences don't play nice together. What are we talking about? refs naming? both? I can have ./refs/sha256:5b58496 but what does that mean that the name will map to blobs/sha256-5b58496? AFAICT the content of that refs which is a descriptor will give us the location in the blobs dirs, not the ref file name.

Third:

$ cd example.com/app/
$ find .

The above (with app/) isn't making clear that an image-layout can carry more than just one app with:

./refs/vbatts-app:latest
./refs/alpine:1.0
./refs/ubuntu:sha256-5bfdjsldf

/cc @vbatts @stevvooe @philips

The text was updated successfully, but these errors were encountered:

runcom · 2016-07-21T13:45:34Z

I believe we need to split the refs and blobs naming paragraph into two pieces - one for refs and one for blobs because clearly the two have different naming restrictions.

wking · 2016-07-21T14:37:42Z

On Thu, Jul 21, 2016 at 06:44:19AM -0700, Antonio Murdaca wrote:

This sentence is wrong because I can have colons : into the refs
subdirectory to say ./refs/vbatts-app:latest

Colons are not portable, and we map them to hyphens 1. If folks
want more characters in the ref names, I suggest we have Unicode names
that are encoded to UTF-8 and then to filesystem-safe base64 2.

Hash algorithm identifiers containing the colon : will be converted to the hyphen -. For example sha256:5b will map to the layout blobs/sha256-5b.
these two sentences don't play nice together. What are we talking
about? refs naming? both?

This is just about the blob filename. Splitting it into its own
paragraph would help clarify that.

AFAICT the content of that refs which is a descriptor will give us
the location in the blobs dirs, not the ref file name.

This is correct.

$ cd example.com/app/
$ find .
The above (with app/) isn't making clear that an image-layout can carry more than just one app with:
./refs/vbatts-app:latest
./refs/alpine:1.0
./refs/ubuntu:sha256-5bfdjsldf

I'm fine adding different app examples here, although I'd recommend
either avoiding colons or base64-encoding the ref names.

runcom · 2016-07-21T15:34:33Z

 Hash algorithm identifiers containing the colon : will be converted to the hyphen -. For example sha256:5b will map to the layout blobs/sha256-5b.
these two sentences don't play nice together. What are we talking
about? refs naming? both?
This is just about the blob filename. Splitting it into its own
paragraph would help clarify that.

it's still misleading - what does that actually mean? what are we talking about? blobs naming? who does the conversion? is it really needed to say that? can't we just say blobs are named ALGO-SUM?
The second sentence is also misleading, where is the case someone come up with `sha256:5b? I guess from a manifest blob, yes, correct, but that sentence right there isn't saying it explicitly, I'm just guessing it.

Not sure about the colon story and base64 - I'd love maintainers to chime in

wking · 2016-07-21T16:21:08Z

On Thu, Jul 21, 2016 at 08:34:36AM -0700, Antonio Murdaca wrote:

Hash algorithm identifiers containing the colon : will be
converted to the hyphen -. For example sha256:5b will map to the
layout blobs/sha256-5b.
these two sentences don't play nice together. What are we
talking about? refs naming? both?
This is just about the blob filename. Splitting it into its own
paragraph would help clarify that.
it's still misleading - what does that actually mean? what are we
talking about? blobs naming? who does the conversion?

The CAS consumer says “get me ‘sha256:5b…’”, and the CAS engine says
“with image-layout 1.0.0, ‘sha256:5b…’ will be stored in
‘blobs/sha256-5b…’” and fetches and returns that content 1.

is it really needed to say that? can't we just say blobs are named
ALGO-SUM?

You could do that too. I don't think it really matters.

The second sentence is also misleading, where is the case someone
come up with `sha256:5b? I guess from a manifest blob, yes, correct,
but that sentence right there isn't saying it explicitly, I'm just
guessing it.

Who cares where sha256:5b… comes from? Both sentences are just “If
you have a blob who's SHA-256 hash has a hex digest 5b…, the name for
that blob is sha256:5b… and the image-layout 1.0.0 path for that blob
is blobs/sha256-5b…”. It doesn't matter if it's a manifest, or a
layer, or a PNG, or whatever.

Consumers will know what sha256:5b… should be from the context of
wherever they got the name (e.g. from a descriptor that also carries
mediaType, or from an email saying “I've just cut a new release…” or
“Here's a cute cat picture”).

runcom · 2016-07-21T16:28:48Z

What's your saying is ok with with as I understand most of the concept and stuff you've highlighted - still I think that's not really clear in those sentences

wking · 2016-07-21T16:36:15Z

On Thu, Jul 21, 2016 at 09:28:49AM -0700, Antonio Murdaca wrote:

… still I think that's not really clear in those sentences

I expect PRs are welcome ;).

stevvooe · 2016-07-21T22:44:50Z

Colons should not be allowed.

The example refs are completely invalid. The following:

./refs/vbatts-app:latest
./refs/alpine:1.0
./refs/ubuntu:sha256-5bfdjsldf

Should be:

./refs/latest
./refs/1.0
./refs/sha256-5bfdjsldf

Otherwise, we completely break windows compatibility, as colons are used for alternate data streams.

runcom · 2016-07-21T22:47:26Z

So there's no way to have multiple images (images as in Ubuntu, Alpine, Debian, Fedora) in a single image layout?!

wking · 2016-07-21T22:48:50Z

On Thu, Jul 21, 2016 at 03:44:51PM -0700, Stephen Day wrote:

Colons should not be allowed.
…
Otherwise, we completely break windows compatibility, as colons are
used for alternate data streams.

This particular bit (or at least, one solution to it) has been spun
off into #174.

stevvooe · 2016-07-21T23:38:54Z

So there's no way to have multiple images (images as in Ubuntu, Alpine, Debian, Fedora) in a single image layout?!

Rather extreme conclusion and tone. Maybe try taking a deep breath before replying? 🐹

The refs simply aren't a tool for naming. They work more like tags and are separated in a similar manner to git. refs provides root references, from which an a set of blobs can be accessed. I'm not even sure that refs will have a one to one mapping with tags (the above example was great).

The path name of a ref must be compatible with as many filesystem and archive formats as possible.

Let's look at alternative example to better understand the future role of refs (not a proposal):

refs/images -> sha256:xxxx, type=some naming thing
blobs/sha:xxxx ->
  {
    "images": [
       { # a descriptor!
         "name": "alpine:latest",
         "digest": "..."
       }
    ]
  }

This part of the specification still needs to be worked out carefully. There are several considerations:

Names need to be compatible across image layout, dns and other discovery systems.
Image Layout needs to support several images from possibly different repositories.
The trust model for naming must be clarified (how does one trust the names in a tar file?). File names aren't a very good authority if you can't verify the integrity of the underlying storage.

wking · 2016-07-22T03:33:24Z

On Thu, Jul 21, 2016 at 04:38:55PM -0700, Stephen Day wrote:

Image Layout needs to support several images from possibly
different repositories.

This seems trivial, since you can drop all the blobs into the same
image-layout bucket, and then pick whatever you like as the refs
pointing into those blobs. The only possible issue I see is if you
want to specify a standard ref-name template for a particular image
name, and I think #174 covers any possible concerns there.

wking · 2016-07-22T04:04:25Z

On Thu, Jul 21, 2016 at 04:38:55PM -0700, Stephen Day wrote:

The trust model for naming must be clarified…

Spun off into #176.

runcom · 2016-07-22T05:56:12Z

Rather extreme conclusion and tone. Maybe try taking a deep breath before replying? 🐹

Ha sorry Stephen, that didn't sound that bad when I wrote it and thanks for explaining that. I was sure that was something image-spec already had and I was just surprised to learn it doesn't fully support that.

runcom · 2016-07-22T06:52:24Z

Image Layout needs to support several images from possibly different repositories.

and this is what I was looking for, thanks

stevvooe · 2016-07-22T18:34:13Z

@runcom No worries.

This is an active area that we need to figure out.

Another possibility is to support a hierarchy in refs. Following from the example above, we'd have something like this:

./refs/vbatts-app/latest
./refs/alpine/1.0
./refs/ubuntu/sha256-5bfdjsldf

This opens the question of whether or not we specify structure in the naming standard.

runcom · 2016-07-22T18:40:04Z

@stevvooe what about having an file which indexes names?

$  cat ./names #ugly
{"busybox":{"refs":"latest"}, "alpine":{"refs":"v1.0"}}

wking · 2016-07-22T18:41:05Z

On Fri, Jul 22, 2016 at 11:34:14AM -0700, Stephen Day wrote:

Another possibility is to support a hierarchy in refs. Following
from the example above, we'd have something like this:
./refs/vbatts-app/latest
./refs/alpine/1.0
./refs/ubuntu/sha256-5bfdjsldf

This is going to be difficult to support portably if we don't have
#174 or special-case path-separator conversion rules to patch around /
on POSIX vs. Windows filesystems.

This opens the question of whether or not we specify structure in
the naming standard.

If “the naming standard” is for signed name-assertions (#176) it seems
completely orthogonal to refs.

stevvooe · 2016-07-22T18:58:55Z

@runcom The makes merging more complicated. Let's say you have two directories with an image layout in them. With the json index approach, it would require special code to merge the json files. If we can define path semantics, the merge can be done with a naive cp -r command.

Albeit, the approach of having a file with the name index provides much better security guarantees, since the set can be verified.

wking · 2016-07-22T19:04:38Z

On Fri, Jul 22, 2016 at 11:58:55AM -0700, Stephen Day wrote:

Albeit, the approach of having a file with the name index provides
much better security guarantees, since the set can be verified.

I'm still missing the need for (or ability to have) security on
mutable refs. If you have a name index that's getting signed, isn't
that a CAS blob?

stevvooe · 2016-07-22T19:31:54Z

@wking Users interact with names. They need to be able to trust that when they say foo, they actually get foo and not something an attacker wants for foo.

wking · 2016-07-22T19:56:27Z

On Fri, Jul 22, 2016 at 12:31:55PM -0700, Stephen Day wrote:

@wking Users interact with names

Agreed, but names != refs. You say “please unpack foo”, and
oci-image-tool (or whatever) says (#88):

Let me look for mutable refs claiming to be foo (this might be
refsEngine.Get("foo"), but could be other things too) and get a
descriptor.
Let me walk the CAS tree that descriptor references looking for a
signed name assertion (Signed name-assertion objects #176).
Does that named.blob assert the name is ‘foo’?
Do I trust any of the signatures in the signed.blob referencing
that named.blob?
Unpack.

The ‘foo’ ref is just a hint for finding the CAS root blob, not part
of the security. It's vulnerable to denial-of-service sorts of
attacks if your ref engine hosts misleading ‘foo’ instances pointing
to CAS trees that you don't end up trusting, so some write security
is in order for public ref stores. But code interacting with
image-layout is the ref store, so I don't think DoS attacks are a
concern there. And I don't see how you'd address DoS ref attacks with
only refs and blobs to work with anyway :p.

stevvooe · 2016-07-22T20:15:01Z

@wking Naive code generally isn't that careful. Drawing the parallel here is dangerous, as programmers will make assumptions.

We can find a solution that is both simple and secure, as long as we define the requirements carefully.

wking · 2016-07-22T20:28:56Z

On Fri, Jul 22, 2016 at 01:15:01PM -0700, Stephen Day wrote:

We can find a solution that is both simple and secure, as long as we
define the requirements carefully.

I guess I'll wait and see when you file that proposal^ ;).

philips · 2016-08-24T14:04:22Z

I am going to take this one and clarify that refs should not include an image name but simply the "tags".

runcom · 2016-08-24T14:07:19Z

I am going to take this one and clarify that refs should not include an image name but simply the "tags".

is this just temporary or what? did we move from refs contains different (unrelated) images also?

vbatts · 2016-08-29T15:05:14Z

@runcom it's more a clarification. After in-person conversation, it was clarified that the refs were intended to be most correlated to the tag current docker image format.
While it could be possible to shove desperate images into a single image-layout, it leads to the kind of confusion you encountered. So perhaps there is a super structure needed for a multi-image layout. (to that end hard-links and/or symlinks ought to be supported for deduplicating the blobs/objects).

runcom · 2016-08-29T15:14:24Z

@runcom it's more a clarification. After in-person conversation, it was clarified that the refs were intended to be most correlated to the tag current docker image format.
While it could be possible to shove desperate images into a single image-layout, it leads to the kind of confusion you encountered. So perhaps there is a super structure needed for a multi-image layout. (to that end hard-links and/or symlinks ought to be supported for deduplicating the blobs/objects).

makes total sense, thx for the explanation @vbatts

Based on the confusion from Antonio in the image-layout in opencontainers#173 we are going to move everything that was in `refs` to `refs/tags`. This does a few things: 1. It makes it clearer that this layout is for a single "image name" instead of allowing for lots of different refs. 2. It matches the layout of git so it likely matches existing expectations. This was also discussed during the OCI F2F and decided to be a reasonable idea. Signed-off-by: Brandon Philips <brandon.philips@coreos.com>

philips · 2016-08-31T02:15:30Z

Trying again with this: #225

wking mentioned this issue Jul 21, 2016

Base64 UTF-8 for image-layout ref names? #174

Closed

wking mentioned this issue Jul 22, 2016

Name compatibility across discovery systems #175

Closed

wking mentioned this issue Jul 22, 2016

Signed name-assertion objects #176

Closed

philips added this to the v0.5.0 milestone Aug 24, 2016

philips self-assigned this Aug 24, 2016

philips mentioned this issue Aug 29, 2016

image-layout: move refs to refs/tags #219

Closed

philips closed this as completed Aug 31, 2016

wking mentioned this issue Oct 23, 2016

Define structure of signatures in OCI #400

Closed

3 tasks

wking mentioned this issue Nov 4, 2016

name-assertion: Add a name-assertion type #445

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Misleading language in image-layout.md #173

Misleading language in image-layout.md #173

runcom commented Jul 21, 2016

runcom commented Jul 21, 2016

wking commented Jul 21, 2016 •

edited

Loading

runcom commented Jul 21, 2016

wking commented Jul 21, 2016

runcom commented Jul 21, 2016

wking commented Jul 21, 2016

stevvooe commented Jul 21, 2016

runcom commented Jul 21, 2016

wking commented Jul 21, 2016

stevvooe commented Jul 21, 2016

wking commented Jul 22, 2016

wking commented Jul 22, 2016

runcom commented Jul 22, 2016 •

edited

Loading

runcom commented Jul 22, 2016

stevvooe commented Jul 22, 2016

runcom commented Jul 22, 2016 •

edited

Loading

wking commented Jul 22, 2016

stevvooe commented Jul 22, 2016

wking commented Jul 22, 2016

stevvooe commented Jul 22, 2016

wking commented Jul 22, 2016

stevvooe commented Jul 22, 2016

wking commented Jul 22, 2016

philips commented Aug 24, 2016

runcom commented Aug 24, 2016

vbatts commented Aug 29, 2016

runcom commented Aug 29, 2016

philips commented Aug 31, 2016

Misleading language in image-layout.md #173

Misleading language in image-layout.md #173

Comments

runcom commented Jul 21, 2016

runcom commented Jul 21, 2016

wking commented Jul 21, 2016 • edited Loading

runcom commented Jul 21, 2016

wking commented Jul 21, 2016

runcom commented Jul 21, 2016

wking commented Jul 21, 2016

stevvooe commented Jul 21, 2016

runcom commented Jul 21, 2016

wking commented Jul 21, 2016

stevvooe commented Jul 21, 2016

wking commented Jul 22, 2016

wking commented Jul 22, 2016

runcom commented Jul 22, 2016 • edited Loading

runcom commented Jul 22, 2016

stevvooe commented Jul 22, 2016

runcom commented Jul 22, 2016 • edited Loading

wking commented Jul 22, 2016

stevvooe commented Jul 22, 2016

wking commented Jul 22, 2016

stevvooe commented Jul 22, 2016

wking commented Jul 22, 2016

stevvooe commented Jul 22, 2016

wking commented Jul 22, 2016

philips commented Aug 24, 2016

runcom commented Aug 24, 2016

vbatts commented Aug 29, 2016

runcom commented Aug 29, 2016

philips commented Aug 31, 2016

wking commented Jul 21, 2016 •

edited

Loading

runcom commented Jul 22, 2016 •

edited

Loading

runcom commented Jul 22, 2016 •

edited

Loading