Custom layer immediate and draped rendering on globe and terrain #12182

akoylasar · 2022-08-23T09:00:34Z

This PR consolidates the changes from #12143 and #11996 to showcase both immediate and draped rendering modes on globe and proposes new necessary APIs.

Note that it might also be necessary to expose an API to allow clients to adjust near and far distances for the camera.

custom-layer-globe.mov

Fixes #11177. This PR requires API required to implement animated wind demo presented in #11996 (comment), but it doesn't include wind demo files.

TODO:

Fix globe to mercator transition in custom-layer-globe.html

Launch Checklist

briefly describe the changes in this PR
include before/after visuals or gifs if this PR includes visual changes
write tests for all new functionality
~~- [ ] document any changes to public APIs~~ Experimental API.
~~- [ ] post benchmark scores~~ N/A
manually test the debug page
~~- [ ] tagged @mapbox/map-design-team @mapbox/static-apis if this PR includes style spec API or visual changes~~
apply changelog label ('bug', 'feature', 'docs', etc) or use the label 'skip changelog'
~~- [ ] add an entry inside this element for inclusion in the mapbox-gl-js changelog: <changelog></changelog>~~

sienki-jenki · 2022-11-24T08:40:56Z

Hey @akoylasar , could you give us a small update on when/if this is going to be merged? I would be thankful 🙏

beifeizhou · 2022-12-02T16:51:44Z

This feature looks great!

May I ask you whether it is possible to load a 3d model around the earth (e.g. ISS station) by setting up the coordinate and the altitude?

mpiannucci · 2023-01-04T21:36:07Z

This is awesome, cant wait to see this merged

akoylasar · 2023-01-05T09:33:33Z

Hey @akoylasar , could you give us a small update on when/if this is going to be merged? I would be thankful 🙏

Hi @sienki-jenki for now there aren't any plans to merge this but if that changes we will keep you updated.

akoylasar · 2023-01-05T09:35:47Z

This feature looks great!

May I ask you whether it is possible to load a 3d model around the earth (e.g. ISS station) by setting up the coordinate and the altitude?

yes it should be possible.

zakjan · 2023-01-05T09:38:15Z

Hi @akoylasar, Is this going to allow upgrading Mapbox integration with deck.gl to support globe, or is there more to that?

akoylasar · 2023-01-17T16:34:18Z

noticed an issue where for some of the tiles the custom layer seems to not get draped into the corresponding offscreen drape texture:

initially I thought that for the said tiles the renderToTile callback might have been left out uninvoked. But as seen from the screenshot above this wasn't the case. Further investigating this.

UPDATE: Some debugging revealed that for the "invisible" tiles above when renderToTiles is invoked stencil test is enabled and is failing causing the fragments rendered by client code to be dropped. It's still not clear why for some of the tiles stencil test is enabled while for other it's not. Further investigating.

karimnaaji

@akoylasar a first pass at reviewing this, but it's looking great already

src/geo/lng_lat.js

karimnaaji · 2023-01-17T23:26:06Z

debug/satellites-custom-layer.js

+            this.updateBuffers();
+
+            gl.enable(gl.DEPTH_TEST);
+            if (globeMatrix !== null) { // globe projection


For discussion: I wonder if there's a better way on the API of custom layer to split between projections. I could imagine two more options:

add a specific value to express which projection we're in (e.g. render(gl, projection, matrices, ...)

have another call for this projection so related code is contained within, e.g. renderGlobe

I find it a bit counter-intuitive to rely on one of the input parameter not being set, but also a bit fragile, if we internally change the interface of this function to not return null,

mapbox-gl-js/src/geo/transform.js

Line 1652 in ad9c713

globeToMercatorMatrix(): ?Array<number> {

we would break such an API and also client-side code.

If you still prefer that design, I'd only suggest to ensure that function always return null under these circumstances with some small tests covering that.

@karimnaaji that's a very good point! There was an implicit agreement with @astojilj that we could always get back to these (i.e. immediate mode related APIs) later as they're experimental anyways. I think we could go with the approach you suggested.

renderGlobe seems better. There is a need to pass additional information, globe matrix, mercatorMatrix (projection matrix passed during globe projection has different z scale from mercator projection matrix used in mercator projection, due to pixelsPerMeter usage in worldToCamera).

mapbox-gl-js/src/geo/transform.js

Lines 1849 to 1875 in 55eab7c

const zUnit = this.projection.zAxisUnit === "meters" ? pixelsPerMeter : 1.0;

const worldToCamera = this._camera.getWorldToCamera(this.worldSize, zUnit);

const cameraToClip = this._camera.getCameraToClipPerspective(this._fov, this.width / this.height, this._nearZ, this._farZ);

// Apply center of perspective offset

cameraToClip[8] = -offset.x * 2 / this.width;

cameraToClip[9] = offset.y * 2 / this.height;

let m = mat4.mul([], cameraToClip, worldToCamera);

if (this.projection.isReprojectedInTileSpace) {

// Projections undistort as you zoom in (shear, scale, rotate).

// Apply the undistortion around the center of the map.

const mc = this.locationCoordinate(this.center);

const adjustments = mat4.identity([]);

mat4.translate(adjustments, adjustments, [mc.x * this.worldSize, mc.y * this.worldSize, 0]);

mat4.multiply(adjustments, adjustments, getProjectionAdjustments(this));

mat4.translate(adjustments, adjustments, [-mc.x * this.worldSize, -mc.y * this.worldSize, 0]);

mat4.multiply(m, m, adjustments);

this.inverseAdjustmentMatrix = getProjectionAdjustmentInverted(this);

} else {

this.inverseAdjustmentMatrix = [1, 0, 0, 1];

}

// The mercatorMatrix can be used to transform points from mercator coordinates

// ([0, 0] nw, [1, 1] se) to GL coordinates.

this.mercatorMatrix = mat4.scale([], m, [this.worldSize, this.worldSize, this.worldSize / pixelsPerMeter, 1.0]);

Edit: prerender needs the same and prerenderGlobe doesn't sound correct. Refactoring to support other projections and updating mercator and globeToMercator matrix to handle difference in z scaling.

src/geo/lng_lat.js

karimnaaji

LGTM % CI

karimnaaji · 2023-01-18T21:51:14Z

src/render/draw_custom.js

-    if (painter.transform.projection.unsupportedLayers && painter.transform.projection.unsupportedLayers.includes("custom")) {
+    if (painter.transform.projection.unsupportedLayers && painter.transform.projection.unsupportedLayers.includes("custom") &&
+        !(painter.terrain && (painter.terrain.renderingToTexture || painter.renderPass === 'offscreen') && layer.isLayerDraped())) {
        warnOnce('Custom layers are not yet supported with non-mercator projections. Use mercator to enable custom layers.');
        return;
    }


Should this error message be updated to note that globe support? I also think the second condition should only early-return, not output an error message:

Suggested change

if (painter.transform.projection.unsupportedLayers && painter.transform.projection.unsupportedLayers.includes("custom")) {

if (painter.transform.projection.unsupportedLayers && painter.transform.projection.unsupportedLayers.includes("custom") &&

!(painter.terrain && (painter.terrain.renderingToTexture || painter.renderPass === 'offscreen') && layer.isLayerDraped())) {

warnOnce('Custom layers are not yet supported with non-mercator projections. Use mercator to enable custom layers.');

return;

}

if (painter.terrain && !painter.terrain.renderingToTexture && painter.renderPass !== 'offscreen' && layer.isLayerDraped()) {

// When terrain is enabled, don't process custom draped layers unless we're in the offscreen pass

return;

}

if (painter.transform.projection.unsupportedLayers && painter.transform.projection.unsupportedLayers.includes("custom")) {

warnOnce('Custom layers are not yet supported with this projection. Use mercator or globe to enable usage of custom layers.');

return;

}

Thanks. Updated comment. Custom layer draping follows the order of other layers and draping happens within the translucent pass.

astojilj · 2023-01-23T17:13:08Z

Transition mercator to globe and back in the demo:

Untitled.mov

karimnaaji

One suggestion up for discussion, but nothing blocking. LGTM

karimnaaji · 2023-01-23T21:21:18Z

src/style/style_layer/custom_style_layer.js


-type CustomRenderMethod = (gl: WebGLRenderingContext, matrix: Array<number>) => void;
+type CustomRenderMethod = (gl: WebGLRenderingContext, matrix: Array<number>, projection?: ProjectionSpecification, projectionToMercatorMatrix?: Array<number>, projectionToMercatorTransition?: number) => void;


Following a bit the conversation from #12182 (comment), and since we're adding a few new parameters in that interface, it might make sense to wrap it under a single variable projectionParams so changes don't break client-side code if we need to remove, add or re-order new parameters to that interface. It would become:

type CustomRenderMethod = (gl: WebGLRenderingContext, matrix: Array<number>, projectionParams: ProjectionParams) => void;

export type ProjectionParams = { projection?: ProjectionSpecification, projectionToMercatorMatrix?: Array<number>, projectionToMercatorTransition?: number };

this can be implemented in a follow-up PR along with the fix for #12182 (comment)

astojilj · 2023-01-24T05:40:39Z

src/geo/lng_lat.js

@@ -2,6 +2,7 @@

 import {wrap} from '../util/util.js';
 import LngLatBounds from './lng_lat_bounds.js';
+import {GLOBE_RADIUS, globeMetersToEcef, latLngToECEF} from '../geo/projection/globe_util.js';


Unit test fail on circular dependency here globe_utils -> mercator_coordinate -> lng_lat -> globe_utils.

akoylasar · 2023-01-24T10:06:25Z

there is one issue where a discontinuity occurs during the transition from globe to mercator as seen in the video below:

discontin.mov

this issue is particularly noticeable closer to the poles.

However to speed things up the fix for this issue will be delivered in a follow-up PR. cc @astojilj @karimnaaji

CustomStyleLayer API extended to support draping with new methods: export type CustomLayerInterface = { ... // misusing MercatorCoordiate's xyz for tile id (x, y, z) where wrap is baked into x. renderToTile: ?(gl: WebGLRenderingContext, tileId: MercatorCoordinate) => void, // return true only for frame when content has changed - otherwise, all the terrain // render cache would be invalidated and redrawn causing huge drop in performance. shouldRerenderTiles: ?() => boolean, } Using WebGL Wind demo code, for simplicity of use, instead of adding submodule, copied dist code. Full code is available at https://github.com/mapbox/webgl-wind/tree/astojilj-draping-support

In globe projection, mercator matrix z scale was incorrect, disabling possibility for transition. Transform.mercatorMatrix changed in order to produce the expected mercator matrix, even in globe (zoom 5->6) transition, and then the same compensated in globe to mercator matrix. updated satellites_custom_layer.js to show transition and satellites in mercator projection.

mourner

Sorry for a late review, just a few nits we can address later.

mourner · 2023-01-24T13:16:01Z

src/geo/lng_lat.js

+        const ecef = latLngToECEF(this.lat, this.lng, radius);
+        return [ecef[0], ecef[1], ecef[2]];


Nit: latLngToECEF already returns a new 3-item array, there seems to be no purpose in making another one.

mourner · 2023-01-24T13:21:37Z

src/geo/projection/globe_util.js

@@ -68,6 +67,10 @@ const GLOBE_LOW_ZOOM_TILE_AABBS = [
    new Aabb([0, 0, GLOBE_MIN], [GLOBE_MAX, GLOBE_MAX, GLOBE_MAX])  // x=1, y=1
 ];

+export function globeMetersToEcef(d: number): number {
+    return d * mercatorZfromAltitude(1, 0.0) * 2.0 * GLOBE_RADIUS * Math.PI;


Nit: we could simplify this. mercatorZfromAltitude(1, 0) = 1 / earthCircumference = 1 / 2 * Math.PI * earthRadius, so we would have:

return d * GLOBE_RADIUS / earthRadius;

mourner · 2023-01-24T13:26:28Z

src/render/draw_custom.js

+            if (painter.transform.projection.name === "globe") {
+                prerender.call(implementation, context.gl, painter.transform.customLayerMatrix(), painter.transform.getProjection(), painter.transform.globeToMercatorMatrix(),  globeToMercatorTransition(painter.transform.zoom));
+            } else {
+                prerender.call(implementation, context.gl, painter.transform.customLayerMatrix());


If I'm reading this right, for non-globe projections, this line will be called twice... A mistake probably?

Yes, this one would be a regression. We need to address this before the release.

Sorry for a late review, just a few nits we can address later.

I will create a bugfix PR for this immediately.

Xin-jj · 2023-06-15T06:31:46Z

I have attempted to create a custom layer that uses a PNG image as a data source, but this layer does not drop on the terrain. Is there any optimization available for such custom layers in this project?

mourner mentioned this pull request Aug 31, 2022

[PoC] Globe view #10784

Closed

akoylasar force-pushed the fouad/custom-layer-immediate-and-draped branch from 38c4621 to f1e47f7 Compare January 16, 2023 09:58

akoylasar added the skip changelog Used for PRs that do not need a changelog entry label Jan 16, 2023

akoylasar marked this pull request as ready for review January 17, 2023 13:01

akoylasar requested a review from a team as a code owner January 17, 2023 13:01

akoylasar changed the title ~~Custom layer immediate and draped on globe~~ Custom layer immediate and draped rendering on globe Jan 17, 2023

akoylasar added feature 🍏 and removed skip changelog Used for PRs that do not need a changelog entry labels Jan 17, 2023

stepankuzmin requested a review from karimnaaji January 17, 2023 16:09

akoylasar requested a review from astojilj January 17, 2023 16:10

karimnaaji reviewed Jan 17, 2023

View reviewed changes

karimnaaji approved these changes Jan 18, 2023

View reviewed changes

astojilj changed the title ~~Custom layer immediate and draped rendering on globe~~ Custom layer immediate and draped rendering on globe and terrain Jan 20, 2023

karimnaaji approved these changes Jan 23, 2023

View reviewed changes

astojilj added the skip changelog Used for PRs that do not need a changelog entry label Jan 23, 2023

astojilj reviewed Jan 24, 2023

View reviewed changes

astojilj and others added 5 commits January 24, 2023 11:15

dirty quick way to render a geometry with custom layer on globe

9f06ee7

expose necessary API, some clean up

8a604c2

use sphere model

c2ed1b6

WIP combining PRs and changing immediate more rendering demo

1c2368a

akoylasar and others added 15 commits January 24, 2023 11:15

add utility func to lnglat class and also render in mercator

4833451

formatgs

589b19d

rm threejs

65ace46

rm unused funcs

cec55a8

minor

fd26f28

render test for draping

c490786

render test updated

a17bc70

minor update to render test

32af347

update debug page to use pure gl only

70a557d

revert accidentally changed files to main

1e00366

address review remarks, fix incorrect color, disable stencil

dbf9173

disable stencil from gl-js side before invoking renderToTile

835ea69

Lint and Flow fixes

ff45cae

break circular dependency, final minor fixes

0979259

akoylasar force-pushed the fouad/custom-layer-immediate-and-draped branch from 93868a1 to 0979259 Compare January 24, 2023 10:15

akoylasar merged commit 5aed4f6 into main Jan 24, 2023

akoylasar deleted the fouad/custom-layer-immediate-and-draped branch January 24, 2023 10:44

This was referenced Jan 24, 2023

CustomLayerInterface support in globe #12046

Closed

WIP: Custom layer draping support (for globe & terrain) #11996

Closed

mourner reviewed Jan 24, 2023

View reviewed changes

akoylasar mentioned this pull request Jan 24, 2023

remove redundant call to prerender from old code, some review fixes #12539

Merged

10 tasks

stepankuzmin mentioned this pull request Jan 26, 2023

Increase "allowed" value for custom-layer-js/draped render test #12541

Merged

3 tasks

This was referenced Jan 27, 2023

fix discontinuity when transitioning from globe to mercator in custom layer example #12544

Merged

Enable immediate mode rendering for custom layer in globe view #12143

Closed

sienki-jenki mentioned this pull request Feb 1, 2023

Simplify globe custom layer interface #12545

Open

Beilinson mentioned this pull request Sep 14, 2023

[Feat] MapboxLayer / MapboxOverlay integration with globe projection visgl/deck.gl#7920

Open

kallyas mentioned this pull request Jul 17, 2024

[Snyk] Upgrade mapbox-gl from 2.9.2 to 2.15.0 kallyas/safe-courier#139

Merged

yash509 mentioned this pull request Sep 16, 2024

[Snyk] Upgrade: firebase, mapbox-gl, next yash509/DevSecOps-Uber-Clone-Deployment#1

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Custom layer immediate and draped rendering on globe and terrain #12182

Custom layer immediate and draped rendering on globe and terrain #12182

akoylasar commented Aug 23, 2022 •

edited by astojilj

Loading

sienki-jenki commented Nov 24, 2022

beifeizhou commented Dec 2, 2022 •

edited

Loading

mpiannucci commented Jan 4, 2023

akoylasar commented Jan 5, 2023

akoylasar commented Jan 5, 2023

zakjan commented Jan 5, 2023

akoylasar commented Jan 17, 2023 •

edited

Loading

karimnaaji left a comment

karimnaaji Jan 17, 2023

akoylasar Jan 18, 2023

astojilj Jan 23, 2023 •

edited

Loading

karimnaaji left a comment •

edited

Loading

karimnaaji Jan 18, 2023

astojilj Jan 23, 2023

astojilj commented Jan 23, 2023

karimnaaji left a comment

karimnaaji Jan 23, 2023

akoylasar Jan 24, 2023

astojilj Jan 24, 2023

akoylasar commented Jan 24, 2023

mourner left a comment

mourner Jan 24, 2023

mourner Jan 24, 2023

mourner Jan 24, 2023

astojilj Jan 24, 2023

akoylasar Jan 24, 2023

Xin-jj commented Jun 15, 2023

	const zUnit = this.projection.zAxisUnit === "meters" ? pixelsPerMeter : 1.0;
	const worldToCamera = this._camera.getWorldToCamera(this.worldSize, zUnit);
	const cameraToClip = this._camera.getCameraToClipPerspective(this._fov, this.width / this.height, this._nearZ, this._farZ);

	// Apply center of perspective offset
	cameraToClip[8] = -offset.x * 2 / this.width;
	cameraToClip[9] = offset.y * 2 / this.height;

	let m = mat4.mul([], cameraToClip, worldToCamera);

	if (this.projection.isReprojectedInTileSpace) {
	// Projections undistort as you zoom in (shear, scale, rotate).
	// Apply the undistortion around the center of the map.
	const mc = this.locationCoordinate(this.center);
	const adjustments = mat4.identity([]);
	mat4.translate(adjustments, adjustments, [mc.x * this.worldSize, mc.y * this.worldSize, 0]);
	mat4.multiply(adjustments, adjustments, getProjectionAdjustments(this));
	mat4.translate(adjustments, adjustments, [-mc.x * this.worldSize, -mc.y * this.worldSize, 0]);
	mat4.multiply(m, m, adjustments);
	this.inverseAdjustmentMatrix = getProjectionAdjustmentInverted(this);
	} else {
	this.inverseAdjustmentMatrix = [1, 0, 0, 1];
	}

	// The mercatorMatrix can be used to transform points from mercator coordinates
	// ([0, 0] nw, [1, 1] se) to GL coordinates.
	this.mercatorMatrix = mat4.scale([], m, [this.worldSize, this.worldSize, this.worldSize / pixelsPerMeter, 1.0]);


		type CustomRenderMethod = (gl: WebGLRenderingContext, matrix: Array<number>) => void;
		type CustomRenderMethod = (gl: WebGLRenderingContext, matrix: Array<number>, projection?: ProjectionSpecification, projectionToMercatorMatrix?: Array<number>, projectionToMercatorTransition?: number) => void;

		const ecef = latLngToECEF(this.lat, this.lng, radius);
		return [ecef[0], ecef[1], ecef[2]];

Custom layer immediate and draped rendering on globe and terrain #12182

Custom layer immediate and draped rendering on globe and terrain #12182

Conversation

akoylasar commented Aug 23, 2022 • edited by astojilj Loading

Launch Checklist

sienki-jenki commented Nov 24, 2022

beifeizhou commented Dec 2, 2022 • edited Loading

mpiannucci commented Jan 4, 2023

akoylasar commented Jan 5, 2023

akoylasar commented Jan 5, 2023

zakjan commented Jan 5, 2023

akoylasar commented Jan 17, 2023 • edited Loading

karimnaaji left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

astojilj Jan 23, 2023 • edited Loading

Choose a reason for hiding this comment

karimnaaji left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

astojilj commented Jan 23, 2023

karimnaaji left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akoylasar commented Jan 24, 2023

mourner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xin-jj commented Jun 15, 2023

akoylasar commented Aug 23, 2022 •

edited by astojilj

Loading

beifeizhou commented Dec 2, 2022 •

edited

Loading

akoylasar commented Jan 17, 2023 •

edited

Loading

astojilj Jan 23, 2023 •

edited

Loading

karimnaaji left a comment •

edited

Loading