-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
KHR_gaussian_splatting proposal spec #72
base: cesium-native
Are you sure you want to change the base?
Conversation
KHR_gaussian_splatting proposal spec
This looks fine to me @keyboardspecialist, and I'm happy to see it merged into the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@keyboardspecialist this looks pretty good. For now I just have comments on the technical aspects, but I think this could also use a style pass.
### Extension attributes | ||
|
||
`extensions.KHR_gaussian_splatting` may contain the following values: | ||
|
||
| Attributes | Type | Description | Required | | ||
| --- | --- | --- | --- | | ||
| quantizedPositionScale | number | Scale value for dequantizing POSITION attribute values | No, default: `1.0` | | ||
|
||
```json | ||
{ | ||
"extensions": { | ||
"KHR_gaussian_splatting": { | ||
"quantizedPositionScale": 13.228187255859375 | ||
} | ||
} | ||
} | ||
``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is quantizedPositionScale
necessary? Usually with KHR_mesh_quantization
the quantization scale is stored in the node transform.
| Splat Data | glTF Attribute | Accessor Type | Component Type | | ||
| --- | --- | --- | --- | | ||
| Position | POSITION | VEC3 | float | | ||
| Color (Spherical Harmonic 0 (Diffuse) and opacity) | COLOR_0 | VEC4 | unsigned byte normalized | | ||
| Rotation | _ROTATION | VEC4 | float | | ||
| Scale | _SCALE | VEC3 | float | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This table could be formatted more like the table in EXT_mesh_gpu_instancing
| Splat Data | glTF Attribute | Accessor Type | Component Type | | ||
| --- | --- | --- | --- | | ||
| Position | POSITION | VEC3 | float | | ||
| Color (Spherical Harmonic 0 (Diffuse) and opacity) | COLOR_0 | VEC4 | unsigned byte normalized | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does COLOR_0
need to be restricted to VEC4
UNSIGNED_BYTE
(normalized)? Someone may have VEC3
colors (no alpha) or floating point values. I feel like all the type/componentType combinations that glTF supports for vertex colors should be allowed here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Allowing float wouldn't be an issue I don't think. I don't know about omitting alpha. It's a core component of splatting. I would need to see the use case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
True... I'm not sure if there's a use case for opaque splats.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given the conversations we have had with Adobe recently, I agree with @keyboardspecialist. I don't think that one could reasonably omit the alpha channel for splats. While we refer to the lowest-order spherical harmonic as diffuse color, that's not really what is going on there. The real view-dependent color is actually packed with the specular in the entire set of the spherical harmonics, and in that case the opacity is required.
Also, I just want to add that based on the conversations we've had with the community and Adobe, I'm guessing that fully supporting the higher-order spherical harmonics is going to need to be a major piece of our next steps in next year.
} | ||
``` | ||
|
||
### Transforming Gaussian Splat Data for glTF |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This section should be marked as non-normative.
|
||
## Adding Gaussian Splats to Primitives | ||
|
||
As Gaussian splats are defined by a position color, rotation and scale, we both map to existing attributes and define new ones. These are attached to a primitive by defining the `extensions.KHR_gaussian_splatting` property on that mesh. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As Gaussian splats are defined by a position color, rotation and scale, we both map to existing attributes and define new ones. These are attached to a primitive by defining the `extensions.KHR_gaussian_splatting` property on that mesh. | |
As Gaussian splats are defined by a position color, rotation and scale, we both map to existing attributes and define new ones. These are attached to a primitive by defining the `extensions.KHR_gaussian_splatting` property on that primitive. |
|
||
#### Sample | ||
|
||
Basic example shown below. This sample shows adding Guassian splats to the first primitive of a mesh. No |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unfinished sentence?
|
||
### Position Attribute | ||
|
||
It is recommended to quantize to `half float` utilizing `KHR_mesh_quantization` with an `unsigned int` accessor type. If using scaled quantization values, you may set the `quantizedPositionScale` value in `extensions.KHR_gaussian_splatting`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
KHR_mesh_quantization
doesn't support half float
... and unsigned int
doesn't provide any space savings. Wouldn't unsigned short
or unsigned byte
be more appropriate depending on your target quantization bits?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're correct. It's actually currently using unsigned short.
| meshopt configuration | value | | ||
| --- | --- | | ||
| Mode | Attributes | | ||
| Filter | None | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The glTF keys/values should probably be used in these tables, e.g.
| meshopt configuration | value | | |
| --- | --- | | |
| Mode | Attributes | | |
| Filter | None | | |
| meshopt configuration | value | | |
| --- | --- | | |
| `mode` | `"ATTRIBUTES"` | | |
| `filter` | `"NONE"` | |
Clean branch with just KHR_gaussian_splatting proposal spec
Extension allows glTF files to treat point primitives as Gaussian splats.