proposal: specifying compression extensions

[fabrobinet]

As a high-level guidance this proposal reuses some concepts learned from GL APIs and extend them to mesh compression:

• have a common API (here properties) for compression (with GL, that's for textures).
• let different implementations be specified via extensions.

Notes:

• as a concrete example of implementation we will use Open3DGC, this said, the proposal should be flexible for other kinds as well…
• currently, the JSON generated in the converter is NOT synced up with this proposal - yet. Ideally we'll get feedback before implementing.

For mesh compression the entities we want to compress - wether they are vertices or indices - are streams of float, int…. can be interdependent/connected and thus all needed at the same time to be able to decompress, but for more flexibility they will appear as separate streams, each represented as an accessor. Depending on cases, it is the extension itself that should specify the info that is required (for instance if indices are required when decompressing vertices to get connectivity info).

Given that, consistency checks should be made depending on each extension requirements

This proposal covers the compression of:

• animations
• meshes
  note: morphing will come later, and should be heavily based on same principles.

In animations accessors contains all the datas and all streams can be compressed separately without dependency. (not connectivity info needed)

The compressed accessor is just declared like any other accessor...

"accessors": {
    "accessor_0": {
        "bufferView": "bufferView_707",
        "byteOffset": 843648,
        "byteStride" : 12102
        "count": 501,
        "type": 5126
    }
...

It is just when accessing the bufferView that we figure that we have to decompress it:

"bufferView_708": {
    "compression" : {
        "type": "Open3DGC",
        "extras" : {
            "mode" : "ascii"
        }
    }
    "buffer": "compressedBuffer",
    "byteLength": 1000,
    "byteOffset": 0
}

We have introduced the compression property. It hold:

• type of compression
• extras: specific datas to this type of compression

The bufferView indicates the byteOffset and byteLength to be read from compressedBuffer to decompress the datas.

As mentioned earlier meshes need connectivity info, so one potential issue with the above definition would be to attempt to uncompress a datas "too soon", e.g when reading the glTF asset if one attempts to decompress bufferViews regardless of the context.
So to prevent this, we either can:

• specify that bufferView with compression should never be decompressed without context (if not access via a mesh or animation).
• or just add say a autonomous(or whatever better name...) property in the compressionproperty, so that we know if bufferView can be uncompressed right away.

That's for the general idea. Based on feedback, implementation will start and iterations are likely to happen after a first pass...

[fabrobinet]

@kmammou you're the compression expert, please let me know what you think. thanks.

[fabrobinet]

thought about some issues with this design...:

• a high number of bufferViews is likely to be generated. (got some ways around this, but not totally satisfying so far)
• because of dependents datas at decompression time, the target property could be just implemented for on bufferView at a time - which defeat a bit the purpose of that keyword...
  Maybe by declaring a buffer as a placeholder to receive uncompressed datas and specify this buffer as a destination for compressed bufferView we could still preserve good usage of the target property. I'll see if I can come up with something simple to implement this...

[pjcozzi]

Comments on the schema:

"bufferView_708": {
    "compression" : {
        "type": "Open3DGC",
        "extras" : {
            "mode" : "ascii"
        }
    },
    // ...
}

• extras (or really extra) is what we are using for forward-compatiable app-specific metadata, so I do not think we should use it in this context. Perhaps use properties instead. Or maybe:

"bufferView_708": {
    "compression" : {
        "Open3DGC" : {
            "mode" : "ascii"
        }
    },
    // ...

As for when to decompress, we need to keep the client really simple. Perhaps we could have a top-level property that is a dependency graph (created at conversion time) of all the bufferViews to decompress. Then, a bufferView without a compression property could be used to create a GL buffer right away, while the rest need to be decompressed first.

[fabrobinet]

• agreed with first point
• I thought about this, but since it is redundant/convenience info I let that for later. What I had in mind was to have 2 properties compressedAttributes and compressedIndices in primitive to be able easily know upfront what needs to be decompressed.

[fabrobinet]

Thought, we discussed that in issues before, having the Open3DGC property right away breaks a bit the "flow" of the implementation. Because developers needs to retrieve the keys for the object in the first place, so like for light and camera I'd like to add type here so we know what we are looking for.

[pjcozzi]

Can you provide a more complete schema example?

Right now, it is very easy for a client to create GL buffers by just inspecting the glTF buffer views. We should strive to keep it as close to this as possible. Since an asset can have some compressed and some uncompressed buffer views (right?), it sounds like this is going to get messy:

1. Go through buffer views and create GL buffers for uncompressed buffer views. Create animation data structures for uncompressed or compressed buffer views.
2. Go through meshes (and primitives) and create GL buffers for compressed buffer views, being careful not to duplicate any work when different meshes point to the same buffer views.

Step 2 feels error prone (we strive to be fast by default) and a lot of pointer chasing. How can we keep this as streamlined as possible and still allow for the decompression dependencies?

[fabrobinet]

I'd like to resolve this next week. Will propose something as early as possible.

[fabrobinet]

Sorry for the delay, I really meant to continue this proposal earlier…

This new iteration was made taking into account feedback from @pjcozzi (which I fully agreed on).
The main improvements brought by this proposal are:

• allow to uncompress all upfront compressed buffers but also on demand depending on client needs
• unified schema to handle decompression of accessors and meshes
• except for the compression property, doesn’t require change to the current spec

How should a mesh or accessor be processed by the compression codec is another more specific discussion.
This proposal scope is to cover:

• how to declare that some entities are compressed
• declare the kind of compression
• location of compressed and uncompressed data

In the following schema example compression is a root property.
Its goal is to expose compressed entities and where is the compressed data to be used.

• compression properties are ids referring to mesh or accessor.
  Those are the only supported entities that can be referred from the compression property for now (based on other use-cases this could be extended to other types).
• the accessor describes the compressed memory.

“compression”: {
  “mesh0” : {
    “type” : “Open3DGC”,    
    “Open3DGC” : {
      “mode” : “ascii”   
    },
    “accessor”: “accessor20”
  },
  “accessor0” : {
    “type” : “Open3DGC”,    
    “Open3DGC” : {
      “mode” : “ascii”   
    },
    “accessor”: “accessor1”
  }
}

To know “where” to uncompress data, the referred mesh or accessor has to be used.
A mesh(e.g its accessors) or a single accessor always point to raw data.
So they actually provide the location where the data needs to be uncompressed and then used for rendering.

It also means that a buffer can be a target, so uri becomes optional in this specific case.
This said the spec should specify that uri is still required if a buffer isn’t a target.

[fabrobinet]

Thinking more about this, there are 2 issues Open3DGC should be associated with extensions and this proposal assume implicitly software decoding. I'll iterate on this again soon.

[pjcozzi]

Looks promising.

When decompression targets a mesh, what does it look like and how does it interact with the accessor the mesh references?

It also means that a buffer can be a target, so uri becomes optional in this specific case. This said the spec should specify that uri is still required if a buffer isn’t a target.

This isn't clear. Can you provide an example?

[pjcozzi]

@kainino0x have you reviewed this discussion in detail? Can we now close this as duplicate with #398?

[pjcozzi]

I talked with @kainino0x and we are going to continue all mesh compression discussion in #398.

[kainino0x]

As mentioned in the proposal in #398, I don't want to create any sort of set structure for mesh compression extensions.

While the three libraries I've looked at (Open3DGC, OpenCTM, webgl-loader) all work similarly, there could easily be other libraries which would be very difficult to shoehorn into this structure.