image-layout: add an initial image layout spec #94

philips · 2016-05-27T02:35:53Z

In order to build tooling that converts the set of OCI Image objects
into an OCI Runtime object we need an Image Layout that test tools can
be built around.

Longer term this Image Layout could be used to define a "single object
image" that is a tar/zip/etc that could be posted over https/ftp/etc.

The layout comes from several different discussions:

#23
#92

Signed-off-by: Brandon Philips brandon.philips@coreos.com

wking · 2016-05-27T03:00:35Z

image-layout.md

@@ -0,0 +1,45 @@
+## Open Container Initiative Image Layout Specification
+
+The OCI Image Layout is a slash separated layout of OCI content-addressable 


"content-adressable ..."? I think you're missing"objects." or similar.

philips · 2016-05-27T04:59:18Z

Any other @opencontainers/image-spec-maintainers want to chime in?

wking · 2016-05-27T04:59:26Z

image-layout.md

+```
+
+Object names in the refs and blobs MUST NOT include characters outside of the set of "A" to "Z", "a" to "z", the hyphen (-), the colon (:), the dot (.), and the underscore (_).
+The blobs directory MAY contain objects which are not referenced by any of the refs.


Looks like GitHub wants you to quote your single-characters, otherwise the underscore leads to a lot of italics (at least in the /pull/94/files view).

And while I'm generally against explicit charset restrictions, in this case I'm fine with them, because the “real” name (i.e. whatever is used for crypo-verification) is going to have to be stored in the contents of a blob. The refs/{nick} entries are just convenient nicknames.

done, thanks.

The combined spec isn't really part of the image format and introduces new objects that we don't have a schema for. Largely I think that it should be replaced by something like the image layout. opencontainers#94 Signed-off-by: Brandon Philips <brandon.philips@coreos.com>

stevvooe · 2016-05-27T19:28:05Z

image-layout.md

+
+The image layout has two top level directories: 
+
+- "blobs" contains content-addressable blobs


I'm not sure I have an opinion either way, I'll regret not bringing this up: we use the term "blobs" because these are free-form binary data. They are not "objects" because they have no required structure, other than that imposed by their access path (ie, through a descriptor).

Using something like "objects" would imply structure or a header, similar to git.

I'm not sure if we include this distinction in the spec.

On Fri, May 27, 2016 at 12:28:05PM -0700, Stephen Day wrote:

+- "blobs" contains content-addressable blobs

I'm not sure I have an opinion either way, I'll regret not bringing
this up: we use the term "blobs" because these are free-form binary
data. They are not "objects" because they have no required
structure, other than that imposed by their access path (ie, through
a descriptor).

I'm not sure what your comment is in response to, but I'm ok with
“objects” → “blobs” throughout this PR.

I am defending the use of the term "blobs" and wondering if we should enshrine said defense. Up until, we've just used it because that is what we've used, whereas there is a deliberate difference between an "object" and a "blob".

On Fri, May 27, 2016 at 12:35:56PM -0700, Stephen Day wrote:

+- "blobs" contains content-addressable blobs

I am defending the use of the term "blobs" and wondering if we
should enshrine said defense…

How about:

The image layout does not specify an internal structure for blobs;
any semantics must be inferred from the blob content or the context
from which the blob was referenced.

I will add some blobs are opaque language.

On Fri, May 27, 2016 at 1:25 PM W. Trevor King notifications@github.com
wrote:

In image-layout.md
#94 (comment)
:

@@ -0,0 +1,47 @@
+## Open Container Initiative Image Layout Specification
+
+The OCI Image Layout is a slash separated layout of OCI content-addressable objects and human significant references (refs)
+This layout MAY be used in a variety of different transport mechanisms from archive formats (e.g. tar, zip) or used in shared filesystem environments (e.g. nfs) or in networked file fetching systems (e.g. http, ftp, rsync).
+Given an image layout a tool can convert a given ref into a runnable OCI Image Format by finding an appopriate manifest from the manifest list, unpacking the filesystem serializations in the correct order, and then converting the image configuration into an OCI Runtime config.json.
+
+The image layout has two top level directories:
+
+- "blobs" contains content-addressable blobs

On Fri, May 27, 2016 at 12:35:56PM -0700, Stephen Day wrote: > +- "blobs"
contains content-addressable blobs I am defending the use of the term
"blobs" and wondering if we should enshrine said defense…
How about: The image layout does not specify an internal structure for
blobs; any semantics must be inferred from the blob content or the context
from which the blob was referenced.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/opencontainers/image-spec/pull/94/files/f6de720992cbc2db439b91d4bd1442cd44693e91#r64962020,
or mute the thread
https://github.com/notifications/unsubscribe/AACDCHGQigCzyqz8tPTcuP4y3DJm1gbHks5qF1NAgaJpZM4IoHyx
.

wking · 2016-05-27T20:11:36Z

Thinking over the backwards-compat discussion in #95, I think we want
to say something here about how we see that working with this layout.
Basically, we want to make it easy and extremely unambiguous to
identify content following this layout from content following some
other layout.

My guess is that we want to give this format a specific name
(application/vnd.oci.image.layout.v1?) in this PR, but punt exposing
that name to the transport mechanisms (e.g. file-based transports can
put it in a magic header, HTTP-based transport can have have a
/version path, etc.). Or maybe we want to require a ‘version’ file
holding the name, and folks who want to also add a magic header are
free to do so?

philips · 2016-05-27T21:16:16Z

I am fine adding a /oci-layout file with contents like:

{"ociLayoutVersion": "1.0.0"}

wking · 2016-05-27T21:28:45Z

On Fri, May 27, 2016 at 02:16:17PM -0700, Brandon Philips wrote:

I am fine adding a root /oci-layout file with contents like:
{"ociLayoutVersion": "1.0.0"}

Sounds fine to me, although I don't see a point to putting ‘oci’ in
both the filename and JSON content.

philips · 2016-05-31T18:22:45Z

@wking @stevvooe all feedback addressed.

@vbatts can you give this a review?

wking · 2016-05-31T18:43:31Z

image-layout.md

@@ -0,0 +1,55 @@
+## Open Container Initiative Image Layout Specification
+
+The OCI Image Layout is a slash separated layout of OCI content-addressable blobs and human significant [location-addressable](https://en.wikipedia.org/wiki/Content-addressable_storage#Content-addressed_vs._location-addressed) references (refs).


I still don't see much point in the hard-to-define “human significant”, especially since location-addressable is sufficient to distinguish refs from blobs.

vbatts · 2016-05-31T21:18:46Z

Would this be an alternative to the combined layout?

philips · 2016-05-31T21:19:24Z

@vbatts yes, combined has been removed because it is based on a bunch of organically grown schemas that aren't super documented. See #95

wking · 2016-05-31T21:19:50Z

On Tue, May 31, 2016 at 02:18:46PM -0700, Vincent Batts wrote:

Would this be an alternative to the combined layout?

A replacement (see #95).

vbatts · 2016-05-31T21:23:06Z

oh wow. I'll move on from protecting the combined format then, once this is incorporated.

So definitely a shim or docker load --format=oci.v1 .. will be needed.

philips · 2016-05-31T21:25:32Z

@vbatts yes.

philips · 2016-06-08T23:49:36Z

@stevvooe @wking lets discuss companion data on #129

philips · 2016-06-09T01:51:58Z

The only thing blocking this is the continued discussion on some sort of prefix sharding: #94 (comment)

I personally feel like we should punt on the sharding. But, if someone can come up with some benchmarks or other evidence that it is necessary for performance or other reasons I am happy to add the language.

stevvooe · 2016-06-09T02:23:34Z

“Content-addressable violation” warning lights started flashing all
over at my house once I read “companion data” ;).

Yep. That is somewhat the point.

Let's say that you want to put something next to the target:

sha256/aa/aaaaaa/
   data

This allows you to later place something next to it:

sha256/aa/aaaaaa/
   data
   references

Its unverified in the default access path, but it can be handy for optimizations. Leaving the extra path component leaves space to add something here in the future.

The best argument against doing this is that most of this should be in the descriptor or other secured data.

I personally feel like we should punt on the sharding. But, if someone can come up with some benchmarks or other evidence that it is necessary for performance or other reasons I am happy to add the language.

This is a best practice. I do admit, there may be a cargo-cult aspect to it.

wking · 2016-06-09T02:41:17Z

On Wed, Jun 08, 2016 at 06:51:59PM -0700, Brandon Philips wrote:

The only thing blocking this is the continued discussion on some
sort of prefix sharding…

I also think it should be rebased around #111 so it can link to the
new descriptor docs.

stevvooe · 2016-06-09T05:08:04Z

@philips Let's not block on the directory layout.

LGTM

vbatts · 2016-06-09T11:01:49Z

LGTM
Though I feel like the structure will come up again

To match the descriptor property, fixing a typo from 9bb56d8 (image-layout: add an initial image layout spec, 2016-05-26, opencontainers#94), although that was developed in parallel with the descriptor spec. Signed-off-by: W. Trevor King <wking@tremily.us>

For folks who want to diverge as little as possible from things already in image-spec. Downsides to this approach include: * Non-sharded blobs [1], although it's not clear to me that modern filesystems suffer from having many entries in one directory [2]. * Possible duplicate blobs between two layouts. You can address this with symlinks or similar, but you'd need extra tooling to do that. With a single CAS bucket, there's only one place that the blob could be, so deduping is free (but garbage collection becomes more complicated). [1]: opencontainers/image-spec#449 [2]: opencontainers/image-spec#94 (comment)

The combined spec isn't really part of the image format and introduces new objects that we don't have a schema for. Largely I think that it should be replaced by something like the image layout. opencontainers/image-spec#94 Signed-off-by: Brandon Philips <brandon.philips@coreos.com>

philips force-pushed the add-image-layout branch from 730ed2a to 5616a30 Compare May 27, 2016 02:36

philips changed the title ~~image-layout: initial commit~~ image-layout: add an initial image layout spec May 27, 2016

This was referenced May 27, 2016

Outline the transportable objects #23

Closed

layout: implement unpacking and validation #82

Merged

wking reviewed May 27, 2016
View reviewed changes

philips mentioned this pull request May 27, 2016

serialization: remove combined spec #95

Merged

stevvooe reviewed May 27, 2016
View reviewed changes

wking reviewed May 31, 2016
View reviewed changes

vbatts merged commit ecdaaaf into opencontainers:master Jun 9, 2016

stevvooe mentioned this pull request Jun 15, 2016

Define generic directory heirarchy for content addressable objects #140

Closed

5 tasks

wking referenced this pull request in sgotti/oci-store Jul 3, 2016

Initial Commit.

29dd456

This was referenced Jul 21, 2016

Misleading language in image-layout.md #173

Closed

Fix broken oci-image-tool #177

Merged

This was referenced Aug 24, 2016

manifest: layer federation #169

Merged

image layout: move to slash separation for blob storage layout #208

Closed

jonboulle mentioned this pull request Oct 14, 2016

README.md: Add high-level link to optional image-layout #380

Merged

This was referenced Nov 3, 2016

image-layout: Fix 'mediatype' -> 'mediaType' in ref payload #439

Merged

layout: sharding the blob store #449

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

image-layout: add an initial image layout spec #94

image-layout: add an initial image layout spec #94

philips commented May 27, 2016

wking May 27, 2016

philips May 27, 2016

philips commented May 27, 2016

wking May 27, 2016 •

edited

Loading

philips May 27, 2016

stevvooe May 27, 2016

wking May 27, 2016

stevvooe May 27, 2016

wking May 27, 2016

philips May 27, 2016

wking commented May 27, 2016

philips commented May 27, 2016 •

edited

Loading

wking commented May 27, 2016

philips commented May 31, 2016

wking May 31, 2016

vbatts commented May 31, 2016

philips commented May 31, 2016

wking commented May 31, 2016

vbatts commented May 31, 2016

philips commented May 31, 2016

philips commented Jun 8, 2016

philips commented Jun 9, 2016

stevvooe commented Jun 9, 2016

wking commented Jun 9, 2016

stevvooe commented Jun 9, 2016 •

edited by caniszczyk

Loading

vbatts commented Jun 9, 2016 •

edited by caniszczyk

Loading

		@@ -0,0 +1,45 @@
		## Open Container Initiative Image Layout Specification

		The OCI Image Layout is a slash separated layout of OCI content-addressable


		The image layout has two top level directories:

		- "blobs" contains content-addressable blobs

		@@ -0,0 +1,55 @@
		## Open Container Initiative Image Layout Specification

		The OCI Image Layout is a slash separated layout of OCI content-addressable blobs and human significant [location-addressable](https://en.wikipedia.org/wiki/Content-addressable_storage#Content-addressed_vs._location-addressed) references (refs).

image-layout: add an initial image layout spec #94

image-layout: add an initial image layout spec #94

Conversation

philips commented May 27, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

philips commented May 27, 2016

wking May 27, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wking commented May 27, 2016

philips commented May 27, 2016 • edited Loading

wking commented May 27, 2016

philips commented May 31, 2016

Choose a reason for hiding this comment

vbatts commented May 31, 2016

philips commented May 31, 2016

wking commented May 31, 2016

vbatts commented May 31, 2016

philips commented May 31, 2016

philips commented Jun 8, 2016

philips commented Jun 9, 2016

stevvooe commented Jun 9, 2016

wking commented Jun 9, 2016

stevvooe commented Jun 9, 2016 • edited by caniszczyk Loading

vbatts commented Jun 9, 2016 • edited by caniszczyk Loading

wking May 27, 2016 •

edited

Loading

philips commented May 27, 2016 •

edited

Loading

stevvooe commented Jun 9, 2016 •

edited by caniszczyk

Loading

vbatts commented Jun 9, 2016 •

edited by caniszczyk

Loading