Light Semantics

[tparisi]

We need to have a general semantic scheme for lighting, similar to how we deal with matrices and other parameters that get passed to the shader.

[fabrobinet]

@tparisi @RemiArnaud @pjcozzi

We have the 4 kind of lights that COLLADA provides. In the context of glTF these are are level objects that are mostly useful for the ones not relying the provided shaders. Someone using provided shaders could just get all the parameters and have the lighting working.

Here is an example of lights (one of each kind):

    "lights": {
        "EnvironmentAmbientLight": {
            "ambient": {
                "color": [
                    1,
                    1,
                    1
                ]
            },
            "id": "EnvironmentAmbientLight",
            "type": "ambient"
        },

        "directionalLightShape1-lib": {
            "directional": {
                "color": [
                    1,
                    1,
                    1
                ]
            },
            "id": "directionalLightShape1-lib",
            "type": "directional"
        },
        "Omni01-light": {
            "id": "Omni01-light",
            "point": {
                "color": [
                    1,
                    1,
                    1
                ],
                "constantAttenuation": 1,
                "linearAttenuation": 0,
                "quadraticAttenuation": 0
            },
            "type": "point"
        },
        "Omni01-light": {
            "id": "Omni01-light",
            "point": {
                "color": [
                    1,
                    1,
                    1
                ],
                "constantAttenuation": 1,
                "linearAttenuation": 0,
                "quadraticAttenuation": 0
            },
            "type": "point"
        },
        "Fspot01-light": {
            "id": "Fspot01-light",
            "spot": {
                "color": [
                    1,
                    1,
                    1
                ],
                "constantAttenuation": 1,
                "fallOffAngle": 43,
                "fallOffExponent": 0,
                "linearAttenuation": 0,
                "quadraticAttenuation": 0
            },
            "type": "spot"
        },
    }
}

Note: we'll add projection to omni, spot and directional. It is useful for the ones doing shadows.

Just like meshes, lights are referred within the node hierarchy:

This is a node:

        "directionalLight1": {
            "children": [],
            "lights": [
                "directionalLightShape1-lib"
            ],
            "matrix": [
                -0.954692,
                0.218143,
                -0.202428,
                0,
                0.0146721,
                0.713885,
                0.700109,
                0,
                0.297235,
                0.665418,
                -0.684741,
                0,
                148.654,
                183.672,
                -292.179,
                1
            ],
            "name": "directionalLight1"
        }

Lights in first iteration are all considered global - as in COLLADA.
We might want later to restrain the influence of lights to some objects, but I believe it's OK to do the first proposal without this.
So.. Since they are global, they impact all the techniques.

Here is for instance how looks the technique for the light in the famous COLLADA duck:

    "techniques": {
        "technique1": {
            "parameters": {
                "ambient": {
                    "type": "FLOAT_VEC4"
                },
                "diffuse": {
                    "type": "SAMPLER_2D"
                },
                "emission": {
                    "type": "FLOAT_VEC4"
                },
                "light0Color": {
                    "type": "FLOAT_VEC3",
                    "value": [
                        1,
                        1,
                        1
                    ]
                },
                "light0Transform": {
                    "source": "directionalLight1",
                    "type": "FLOAT_MAT4"
                },
                "normal": {
                    "semantic": "NORMAL",
                    "type": "FLOAT_VEC3"
                },
                "normalMatrix": {
                    "semantic": "WORLDVIEWINVERSETRANSPOSE",
                    "type": "FLOAT_MAT3"
                },
                "position": {
                    "semantic": "POSITION",
                    "type": "FLOAT_VEC3"
                },
                "projectionMatrix": {
                    "semantic": "PROJECTION",
                    "type": "FLOAT_MAT4"
                },
                "reflective": {
                    "type": "FLOAT_VEC4"
                },
                "shininess": {
                    "type": "FLOAT"
                },
                "specular": {
                    "type": "FLOAT_VEC4"
                },
                "texcoord0": {
                    "semantic": "TEXCOORD_0",
                    "type": "FLOAT_VEC2"
                },
                "worldViewMatrix": {
                    "semantic": "WORLDVIEW",
                    "type": "FLOAT_MAT4"
                }
            },
            "pass": "defaultPass",
            "passes": {
                "defaultPass": {
                    "instanceProgram": {
                        "attributes": {
                            "a_normal": "normal",
                            "a_position": "position",
                            "a_texcoord0": "texcoord0"
                        },
                        "program": "program_0",
                        "uniforms": {
                            "u_ambient": "ambient",
                            "u_diffuse": "diffuse",
                            "u_emission": "emission",
                            "u_light0Color": "light0Color",
                            "u_light0Transform": "light0Transform",
                            "u_normalMatrix": "normalMatrix",
                            "u_projectionMatrix": "projectionMatrix",
                            "u_reflective": "reflective",
                            "u_shininess": "shininess",
                            "u_specular": "specular",
                            "u_worldViewMatrix": "worldViewMatrix"
                        }
                    },
                    "states": {
                        "blendEnable": false,
                        "cullFaceEnable": true,
                        "depthMask": true,
                        "depthTestEnable": true
                    }
                }
            }
        }
    },

It's important to note that lights need to have a mechanism in place to refer not only the node being rendered, for instance for a directional light we need to get the transform of the node that hold the light.

To solve this I added a source property in parameter like this:

                "light0Transform": {
                    "source": "directionalLight1",
                    "type": "FLOAT_MAT4"
                },

This needs refinement as this is assuming WORLDMATRIX (or MODELMATRIX whatever we will call it eventually).
This would just mean also implicitly that is no source is specified then the node being rendered is the source. (just to extend the logic to what we have already).
I am not totally sold on how this looks right now, we'll iterate post SIGGRAPH about this particular issue.

Another thing to consider will be to support senantic that are not necessarily matrices, for some lights types it could be convenient to have just positions.

[tparisi]

The basic proposal is fine for SIGGRAPH. We'll get basic lights working
that way.

Regarding the light matrices, positions, etc. couldn't we simply add a
semantic property to the parameter?

On Mon, Jul 15, 2013 at 1:38 PM, Fabrice Robinet
notifications@github.comwrote:

@tparisi https://github.com/tparisi @RemiArnaudhttps://github.com/RemiArnaud
@pjcozzi https://github.com/pjcozzi

We have the 4 kind of lights that COLLADA provides. In the context of glTF
these are are level objects that are mostly useful for the ones not relying
the provided shaders. Someone using provided shaders could just get all the
parameters and have the lighting working.

Here is an example of lights (one of each kind):

"lights": {
    "EnvironmentAmbientLight": {
        "ambient": {
            "color": [
                1,
                1,
                1
            ]
        },
        "id": "EnvironmentAmbientLight",
        "type": "ambient"
    },

    "directionalLightShape1-lib": {
        "directional": {
            "color": [
                1,
                1,
                1
            ]
        },
        "id": "directionalLightShape1-lib",
        "type": "directional"
    },
    "Omni01-light": {
        "id": "Omni01-light",
        "point": {
            "color": [
                1,
                1,
                1
            ],
            "constantAttenuation": 1,
            "linearAttenuation": 0,
            "quadraticAttenuation": 0
        },
        "type": "point"
    },
    "Omni01-light": {
        "id": "Omni01-light",
        "point": {
            "color": [
                1,
                1,
                1
            ],
            "constantAttenuation": 1,
            "linearAttenuation": 0,
            "quadraticAttenuation": 0
        },
        "type": "point"
    },
    "Fspot01-light": {
        "id": "Fspot01-light",
        "spot": {
            "color": [
                1,
                1,
                1
            ],
            "constantAttenuation": 1,
            "fallOffAngle": 43,
            "fallOffExponent": 0,
            "linearAttenuation": 0,
            "quadraticAttenuation": 0
        },
        "type": "spot"
    },
}

}

Note: we'll add projection to omni, spot and directional. It is useful for
the ones doing shadows.

Just like meshes, lights are referred within the node hierarchy:

This is a node:

    "directionalLight1": {
        "children": [],
        "lights": [
            "directionalLightShape1-lib"
        ],
        "matrix": [
            -0.954692,
            0.218143,
            -0.202428,
            0,
            0.0146721,
            0.713885,
            0.700109,
            0,
            0.297235,
            0.665418,
            -0.684741,
            0,
            148.654,
            183.672,
            -292.179,
            1
        ],
        "name": "directionalLight1"
    }

Lights in first iteration are all considered global - as in COLLADA.
We might want later to restrain the influence of lights to some objects,
but I believe it's OK to do the first proposal without this.
So.. Since they are global, they impact all the techniques.

Here is for instance how looks the technique for the light in the famous
COLLADA duck:

"techniques": {
    "technique1": {
        "parameters": {
            "ambient": {
                "type": "FLOAT_VEC4"
            },
            "diffuse": {
                "type": "SAMPLER_2D"
            },
            "emission": {
                "type": "FLOAT_VEC4"
            },
            "light0Color": {
                "type": "FLOAT_VEC3",
                "value": [
                    1,
                    1,
                    1
                ]
            },
            "light0Transform": {
                "source": "directionalLight1",
                "type": "FLOAT_MAT4"
            },
            "normal": {
                "semantic": "NORMAL",
                "type": "FLOAT_VEC3"
            },
            "normalMatrix": {
                "semantic": "WORLDVIEWINVERSETRANSPOSE",
                "type": "FLOAT_MAT3"
            },
            "position": {
                "semantic": "POSITION",
                "type": "FLOAT_VEC3"
            },
            "projectionMatrix": {
                "semantic": "PROJECTION",
                "type": "FLOAT_MAT4"
            },
            "reflective": {
                "type": "FLOAT_VEC4"
            },
            "shininess": {
                "type": "FLOAT"
            },
            "specular": {
                "type": "FLOAT_VEC4"
            },
            "texcoord0": {
                "semantic": "TEXCOORD_0",
                "type": "FLOAT_VEC2"
            },
            "worldViewMatrix": {
                "semantic": "WORLDVIEW",
                "type": "FLOAT_MAT4"
            }
        },
        "pass": "defaultPass",
        "passes": {
            "defaultPass": {
                "instanceProgram": {
                    "attributes": {
                        "a_normal": "normal",
                        "a_position": "position",
                        "a_texcoord0": "texcoord0"
                    },
                    "program": "program_0",
                    "uniforms": {
                        "u_ambient": "ambient",
                        "u_diffuse": "diffuse",
                        "u_emission": "emission",
                        "u_light0Color": "light0Color",
                        "u_light0Transform": "light0Transform",
                        "u_normalMatrix": "normalMatrix",
                        "u_projectionMatrix": "projectionMatrix",
                        "u_reflective": "reflective",
                        "u_shininess": "shininess",
                        "u_specular": "specular",
                        "u_worldViewMatrix": "worldViewMatrix"
                    }
                },
                "states": {
                    "blendEnable": false,
                    "cullFaceEnable": true,
                    "depthMask": true,
                    "depthTestEnable": true
                }
            }
        }
    }
},

It's important to note that lights need to have a mechanism in place to
refer not only the node being rendered, for instance for a directional
light we need to get the transform of the node that hold the light.

To solve this I added a source property in parameter like this:

            "light0Transform": {
                "source": "directionalLight1",
                "type": "FLOAT_MAT4"
            },

This needs refinement as this is assuming WORLDMATRIX (or MODELMATRIX
whatever we will call it eventually).
This would just mean also implicitly that is no source is specified then
the node being rendered is the source. (just to extend the logic to what we
have already).
I am not totally sold on how this looks right now, we'll iterate post
SIGGRAPH about this particular issue.

Another thing to consider will be to support senantic that are not
necessarily matrices, for some lights types it could be convenient to have
just positions.

�
Reply to this email directly or view it on GitHubhttps://github.com//issues/93#issuecomment-20987534
.

Tony Parisi tparisi@gmail.com
CTO at Large 415.902.8002
Skype auradeluxe
Follow me on Twitter! http://twitter.com/auradeluxe
Read my blog at http://www.tonyparisi.com/
Learn WebGL http://learningwebgl.com/

Read my book! WebGL, Up and Running
http://shop.oreilly.com/product/0636920024729.do
http://www.amazon.com/dp/144932357X

[fabrobinet]

Yes @tparisi I think so.

[fabrobinet]

To finish this, we need the semantic and thus blocked by: #83

[tparisi]

I am lost in the details and how it relates to issue #83 . I think a face to face with Fabrice and he and I could settle the remaining details. @fabrobinet late next week?

[fabrobinet]

It relates to #83 because we just discussed about the semantic property of the source parameter:
i.e

  "light0Transform": {
                    "source": "directionalLight1",
                    "type": "FLOAT_MAT4"
                    "semantic" : "WORLDVIEW"
                },

[pjcozzi]

Shouldn't the id properties above be name like a glTF node?

        "EnvironmentAmbientLight": {
            "ambient": {
                "color": [
                    1,
                    1,
                    1
                ]
            },
            "id": "EnvironmentAmbientLight",
            "type": "ambient"
        },

[fabrobinet]

I do not repro this issue with dev-5 (in flux) and probably not with dev-4. Yes it should be just that way.:

        "EnvironmentAmbientLight": {
            "ambient": {
                "color": [
                    0.905882,
                    0.886275,
                    0.866667
                ]
            },
            "type": "ambient"
        },

[pjcozzi]

@fabrobinet with or without name? You are showing it without name.

[fabrobinet]

name is not mandatory, but it could be there yes.

[pjcozzi]

Agreed. Thanks.

[fabrobinet]

flagged for coming-next to add semantic.

[fabrobinet]

Actually, lights in the context of glTF could be just represented as parameters for shaders, and we can provide a descriptive object within the details property for the ones willing to regenerate shaders.

[pjcozzi]

Agreed 👍

[fabrobinet]

I have added semantic (to specify which kind of transform we want) along with source for light parameters, but lights still needs a bit of schema simplification, as agreed earlier, the lights property is redundant with the parameters set in the shaders, and its place for high level info is more inside the details property. The section about the convectional names for lights then should be added in the spec for the details property.

[tparisi]

I'm not following... are we saying we would get rid of the light-specific parameters? I don't want to lose that info, it's useful for higher-level engines like Three.js where we can optionally use common techniques like Phong.

[fabrobinet]

We will keep all the parameters within techniques, but will remove lights objects in the node hierarchy because they are not really useful here as have to declared anyway also in the techniques to set the parameters and uniforms.

[tparisi]

If they're not in the node hierarchy, how can they be transformed and animated?

[fabrobinet]

Oops, sorry I didn't meant that :)

The intention here is actually to remove the lights property at the root level of the JSON - but of course not node that hold the transform.
lights hold entries to describes values such as color of the lights, attenuation... but these parameters are eventually redundant in the technique.

[tparisi]

Ah ok, well that makes me feel better. However I don't quite understand what the resulting file would look like. Do you have a quick example? I don't need to see all the technique info, just where the light would be defined in the file, and how it would be referenced as a node. So, pseudo-glTF would suffice...

[fabrobinet]

Based the proposed changes in #361 we extend here the discussion to lights.

1. We can remove the lights properties at the root level of glTF as the ones who directly use the shaders provided by glTF have all the paramaters/values for lights already set within the techniques.
2. For the ones who do not want to use the shaders provided by glTF and re-create their owns, light properties should be specified within the context of a specific extension as explained in details property should be removed in favor of an extension #361 .
   Because glTF is not about specifying lightingModel and depending of input format this information could vary a lot, it is safer to handle through extensions.

As an example for 2 with the converter available on this repo, then we'd have an extension defined within pass such as:

"allExtensions" : ["COLLADA-1.4.1/commonProfile"],
"extensions": {
    "COLLADA-1.4.1/commonProfile": {
        "extras": {
            "doubleSided": false
        },
        "lightingModel": "Blinn",
        "parameters": {
            "ambient" : "ambient",
            "diffuse" : "diffuse",
            "emission" : "emission",
            "reflective" : "reflective",
            "shininess" : "shininess",
            "specular" : "specular",
            "transparency" : "opacity",
            "lights" : {
                "light1" : {
                    "color" : "light1Color",
                    "constantAttenuation": "light1ConstantAttenuation",
                    "linearAttenuation": "light1LinearAttenuation",
                    "quadraticAttenuation" : "light1QuadraticAttenuation"
                }
            }
        }
    },
    "texcoordBindings": {
        "ambient": "TEXCOORD_0",
        "diffuse": "TEXCOORD_0",
        "reflective": "TEXCOORD_0"
    }
},

• the biggest benefit is that glTF core won't be impacted in any way about concepts tied to lights.
• this is consistent with other material properties. (as semantic such as "specular" and "constantAttenuation" point both to actual parameters within techniques)

[fabrobinet]

@tparisi @pjcozzi @RemiArnaud what do you think ?
Also I am not completely sure here about the schema for the extension named COLLADA-1.4.1/commonProfile but we can refine it here #362

[pjcozzi]

All sounds good to me. I am happy to update the schema when we determine the exact details for this extension.

[RemiArnaud]

I�m not sure what the question is.

If this is an extension to the �pass� object, then I would call this �extra� and then underneath have �profile� sections for each extensions
not sure what the �allExtensions' is for, or what 'COLLADA-1.4.1/commonProfile� is referring to exactly.

Regards
� Remi

[tparisi]

I'm sorry I object to this whole change. I personally thought the lights stuff was fine the way it is and now this is getting more COLLADA-like as the design evolves.

Why do we need extensions to handle something as basic as lights?

In practice, a runtime wants to know about the lights, cameras and animations in a scene. Having them in a list at the top level is extremely helpful. For implementations that don't need that, they can just ignore the property.

What I have liked about glTF so far is that we have kept the design practical. Design choices like this move away from practical.

[fabrobinet]

I'm sorry I object to this whole change. I personally thought the lights stuff was fine the way it is and now this is getting more COLLADA-like as the design evolves.

This is actually quite the opposite - LightingModel specification in COLLADA was mostly inherited from the OpenGL fixed pipeline and removing all references about lights has been the trend since OpenGL ES 2.0. So this move would bring glTF in a closer place to modern GL APis than COLLADA.

Why do we need extensions to handle something as basic as lights?

If you use the provided shaders (as Cesium) developers can write based on glTF a completely data-driven engine and do not need to know about high level concepts such as lights to actually perform a rendering that contains lighting.

For the ones who want to re-create shaders (as you do), I find it risky to commit on specifying in lights - as depending on tools... lighting models may vary and we could be very soon limited with only the traditional 4 types of lights (ambient, omni, directional, spot). One could write a glTF exporter right in a tool (3DSMAX...) and export other types of lights that we aren't aware of, and the place to describe such light should be in a specific extension.

In practice, a runtime wants to know about the lights, cameras and animations in a scene. Having them in a list at the top level is extremely helpful. For implementations that don't need that, they can just ignore the property.

I got your point but addressing this could be easy.
We can an add also extension property at the root level of the JSON and instead of directly parsing the lights property you'd have the go through the extension first.

What I have liked about glTF so far is that we have kept the design practical. Design choices like this move away from practical.

Again I get your argument, but it's the exact same debate as if you were going to OpenGL WG and were asking to put back the fixed pipeline because it was practical...

[tparisi]

@fabrobinet @pjcozzi Guys I am coming around to this way of thinking. HOWEVER can we consider changing the naming from "extensions"? This doesn't feel like an extension to me-- extensions are for significant added capability-- and I think the naming will confuse people. It sounds a lot more like a technique or profile or something. In fact, "profiles" seems like a pretty good name for this one. Let's leave "extensions" for true extensions. What do you think?

Secondary naming thing: I think we should not have COLLADA in this name but rather call it something like "Materials-FixedFunction" because this is a generally useful set of information whether or not it is implemented in COLLADA. For my applications I don't care about COLLADA, I just want to make sure I can get the fixed function material and lighting information for simple applications such as viewers or previewing tools.

[fabrobinet]

@tparisi profile is already taken by asset/profile (https://github.com/KhronosGroup/glTF/blob/master/specification/asset.schema.json) but I get your concern about extension that may not be the most natural naming for this and we can think of another name. It's related to the tool/exporter who generated the glTF file... I don't have anything that sounds right on the top of my head right now for this. We'll see if we circle back on pass/profile...

Then yes, we can revise how we will actually name the extension. We can discuss that in #362

[tparisi]

It's a plan.

FWIW I think "profiles" is still OK, it's in a different context.

"extras?"

I will meditate on it, maybe I can think of something?

[pjcozzi]

All sounds good to me.

[pjcozzi]

We can close this when we finish materials in #84.

[pjcozzi]

@tparisi can you quickly review this and close as duplicate with #424 if you agree?

[tparisi]

Closing, moved to common materials extension. Duplicate of #424