Binary glTF

[pjcozzi]

Currently, we have two delivery options:

• glTF JSON points to external binary (geometry, key frames, skins), image, and shaders
• glTF JSON has base64-encoded binary, images, and shaders using a data uri (Format should support references and inline data #68).

These can be mixed and matched, of course.

Base64-encoding increases the file size and requires extra decoding. glTF is commonly criticized for requiring separate requests or extra space due to base64-encoding.

To solve this, I suggest what I've called "Binary glTF", which could become glTF itself or at least a container format. In Binary glTF, we put everything (JSON, .bin, images, shaders) into a single binary blob. We then extract the JSON text from the binary blob with TextDecoder (here's our wrapper), use JSON.parse as usual, and then treat the binary blob as a glTF buffer.

We can also think of this approach as embedding the JSON, images, and shaders in the .bin file.

The Binary glTF format we prototyped is:

magic number, uint32 // 'glTF'
version, uint32
jsonOffset, uint32
jsonLength, uint32
// embedded (or separate) data

Our loader is here.

This required some minor spec additions (which are up for discussion):

• "self" is the buffer that references the binary glTF file.
• shader can have a uri (as usual) or a bufferView so the text can be extracted from the typed array (binary glTF).
• image can have a uri (as usual) or bufferView, mimeType, width, and height, which allows users to create a JavaScript image like this.

File size and number of files

Tested with our aircraft model (we need to test with me).

	dae	glTF	glTF (base64-encoded bin/png/glsl)	Binary glTF
size	1.07 MB (3 files)	802 KB (8 files)	1.03 MB	806 KB
size (gzip)	728 KB	677 KB	706 KB	707 KB

(Not impressed by gzip size for this example)

Binary glTF still supports external resources and embedded base64 ones. For example, I think a common case will be to embed .bin and shaders into a binary glTF and then have external images.

Questions

• Do we want this to become core glTF? Or a container format?
• Any recommended tweaks to the schema?

[pjcozzi]

@fabrobinet what do you think?

[fabrobinet]

I feel like (in some case) storing base64 in binary file is a bit strange (or did I misunderstood?).
I'd prefer considering a fully binary alternative if we go on this road, but I don't feel it's critical for v1.0.

As a post-process on the output from the converter, repackaging as you suggest if you like for cesium is fine, but it would be a layer on top of the converter and a specific way to handle for the client. glTF isn't an inter-exchange format, so it's probably fine. And so the spec wouldn't have to be modified.

I also don't feel like this brings a lot compared with just converting an asset that would be zipped (including all resources, images etc..). I did months ago a export options to a "bundle" which is just a folder. I could refresh that... and just zip it.

[pjcozzi]

Perhaps I didn't explain this correctly...

I'd prefer considering a fully binary alternative if we go on this road, but I don't feel it's critical for v1.0.

This is a full binary alternative. The key insight is that the glTF JSON is in the binary blob and that the self buffer refers to the binary blob so that everything (JSON, .bin, .glsl, images) are packed into a single binary blob without requiring any base64-encoding. Of course, some resources could still be external or base64-encoded (but that is not needed).

I also don't feel like this brings a lot compared with just converting an asset that would be zipped...

Unless I am missing something, this is way better. How will the client unzip? In JavaScript? With this approach, the client receives a typed array and then can do fast operators to extract the JSON and images. See getStringFromTypedArray and loadImageFromTypedArray.

repackaging as you suggest if you like for cesium is fine

I am fine with this (honestly, it gives Cesium an edge since this will be superior to all other approaches) but I want to make sure you understand this proposal before I go this route.

[fabrobinet]

I'd prefer considering a fully binary alternative if we go on this road, but I don't feel it's critical for v1.0.

This is a full binary alternative. The key insight is that the glTF JSON is in the binary blob and that the self buffer refers to the binary blob so that everything (JSON, .bin, .glsl, images) are packed into a single binary blob without requiring any base64-encoding. Of course, some resources could still be external or base64-encoded (but that is not needed).

What I meant by not fully binary is that you embed JSON and - dependending on convertion options - base 64 data right in a binary file. So, the binary ends up being a concatenation of multiple files of different nature right ?.
I am not saying that it is bad, I am just saying that the inner data in the binary blob you create (as the JSON) could be represented more efficiently upfront.

So the question is do we want to standardize a container format for glTF now ?
At this point I am not sure, but i completely get it is useful, I am just not clear if that's a bit too soon since there are other things to explore such as http://msgpack.org/ mentioned here #348 (comment) (not sure why the issue was deleted).

Also I'd like to hear about other opinions about this: @RemiArnaud , @tparisi .. ?

[pjcozzi]

dependending on convertion options - base 64 data right in a binary file

Right, you could do that, but it would be wasteful and unnecessary.

I am not saying that it is bad, I am just saying that the inner data in the binary blob you create (as the JSON) could be represented more efficiently upfront.

Totally agree, but does it matter relative to the geometry/animation/texture payload? I suspect almost never. JSON.parse is also presumably very fast. Is it faster than walking through a custom format in a typed array? Probably not, but I suspect it is way faster than the real bottlenecks of download the asseting, compiling shaders, and uploading vertex data and textures.

Unrelated but the other thing I want to do - at least for my own use - is be able to reference a material/technique/shader that is not actually in the glTF, e.g., I download the material once but use it for 100s of models that I download.

We actually are still considering embedding msgpacked JSON, but didn't get to it yet, because (1) I suspect the wins are minimal and (2) decode happens in JavaScript.

So the question is do we want to standardize a container format for glTF now ?

I am totally cool with waiting post 1.0. This could be a proposed extension (I can write it up) and then we can approve it or not when we are ready to commit. In the meantime, I will be using it in Cesium for something BIG (literally, lol).

[fabrobinet]

dependending on convertion options - base 64 data right in a binary file

Right, you could do that, but it would be wasteful and unnecessary.

So we agree, that's what I called a strange possibility earlier in the thread...
Let's say the packer should be smart and re-encode base64 in real binary within the container in this case within the container and we'd be good for this issue.

Also, I'd suggest a few minor updates in the binary "stub" , so that for future extensibility of the container you do not assume JSON for instance, but instead of you specify a content-type. (so that it provision for msg-pack or whatever binary representation for the future).

Let's give it a bit more thinking, and sync-up with other contributors :)

[tparisi]

@pjcozzi I love this whole thread... but I think I agree with one of @fabrobinet 's earlier comments, namely that it's a bit too soon. Let's finalize 1.0 and put this on the short list for right after.

[pjcozzi]

OK, I will write it up as a proposed extension. That will be good practice anyway and will give me some freedom to easily tweak it as we continue to do development. Thanks for the input.

[fabrobinet]

Sounds good @pjcozzi and thanks for all the work about this !.
So we'll keep this issue open to track the extension proposal then.

[pjcozzi]

I wrote this up as an extension.

https://github.com/KhronosGroup/glTF/blob/new-extensions/extensions/CESIUM_binary_glTF/README.md

I know this isn't 1.0 but I need it for a major feature in Cesium so I needed to at least get it written down.

I understand if we don't have time for feedback on this, but I would like to know if we are going to allow vendor prefixes for niche extensions coming from a single vendor. That is, should this be CESIUM_binary_glTF or just binary_glTF?

I also wrote up this very niche extension:

https://github.com/KhronosGroup/glTF/blob/new-extensions/extensions/CESIUM_RTC/README.md

[pjcozzi]

Also, if we want to completely separate these extensions from glTF for the time-being, I am OK with putting them in a separate Cesium-related repo instead of in a branch here in the official repo. Let me know what you think.

[pjcozzi]

The finishing touches on this are being discussed in the pull request, #400.