Add option to load rtl-text-plugin lazily #8865

arindam1993 · 2019-10-12T02:56:34Z

Addresses #8825.
~~Adds an rtlTextPluginURL option to the map, and map will load the rtlTextPlugin the first time rtl text is encountered~~

Adds a new lazy parameter to setRTLTextPlugin(url: string, cb: Function, lazy: boolean). Its false by default to keep existing behavior, but when set to true, it will cause mapboxgl to lazily load the rtl-text-plugin the first time it encounters rtl-text, it also prevents the rtl-text from rendering until the plugin is downloaded and parsed after which it loads in.

This includes a few more enhancements,

It will now make the network request for the plugin only once, and then distribute the downloaded data to all workers via a blobUrl
the callback function is now invoked only once when the plugin is parsed and ready on all workers, instead of being repeatedly invoked for each worker.

Under the hood, theres a re-write of how the main-thread -> worker communication for the plugin workers. The main thread now has full ownership of the state of the plugin and pushes it to the workers. The workers are reactive and download and parse the plugin once all the data is available.

Launch Checklist

briefly describe the changes in this PR
~~include before/after visuals or gifs if this PR includes visual changes~~
write tests for all new functionality
document any changes to public APIs
~~post benchmark scores~~
manually test the debug page
~~tagged @mapbox/studio and/or @mapbox/maps-design if this PR includes style spec or visual changes~~
~~tagged @mapbox/gl-native if this PR includes shader changes or needs a native port~~

asheemmamoowala

@arindam1993 Can you describe the expected behavior when using this map options?

What I am seeing when loading this URL: http://localhost:9966/debug/rtl-plugin-autoload.html#12.22/27.34633/45.00641, with throttled network or CPU performance (from Chrome dev tools), is that the Arabic text shows up left-to-right, and flashes before being re-rendered correctly.

Would it be possible to hold off that text from being rendered so that it fades in after the plugin is loaded?

Additionally, this API only works with a URL to a separately hosted rtl-text-plugin. This means that developers who package the plugin, or bundle GL-JS differently will not be able to load this plugin on-demand. (cc @kappaxbeta @patfrat)

There's a fair bit of overlap between this PR and #8864. It may make sense to commit #8664 first and then rebase this on top of that so that it can take advantage of the loading/requested state. In my debugging, the plugin was being fetched twice, based on the Devtools network tab.
cc @ansis

arindam1993 · 2019-10-15T03:54:50Z

What I am seeing when loading this URL: http://localhost:9966/debug/rtl-plugin-autoload.html#12.22/27.34633/45.00641, with throttled network or CPU performance (from Chrome dev tools), is that the Arabic text shows up left-to-right, and flashes before being re-rendered correctly.

Would it be possible to hold off that text from being rendered so that it fades in after the plugin is loaded?

Yes, the expected behaviour should be, all LTR text loads as expected, and RTL text loads appears after a slight delay once the plugin finishes downloading.

Additionally, this API only works with a URL to a separately hosted rtl-text-plugin. This means that developers who package the plugin, or bundle GL-JS differently will not be able to load this plugin on-demand. (cc @kappaxbeta @patfrat)

I think this PR shouldn't try to rework the delivery process, but we can make the loading lazy and slightly more efficient.

There's a fair bit of overlap between this PR and #8864. It may make sense to commit #8664 first and then rebase this on top of that so that it can take advantage of the loading/requested state. In my debugging, the plugin was being fetched twice, based on the Devtools network tab.
cc @ansis

Yes, will wait for #8864 to be ready, the plugin being fetched multiple times is the current status-quo though, since each worker requests for the plugin separately. I'll change that in this PR so that the main thread downloads the plugin and then the workers load it via blob url.

ryanhamley · 2019-10-15T18:54:09Z

src/data/bucket/symbol_bucket.js

@@ -413,7 +414,9 @@ class SymbolBucket implements Bucket {
                if (formattedText.containsRTLText()) {
                    this.hasRTLText = true;
                }
-                text = transformText(formattedText, layer, feature);
+                if (this.hasRTLText && globalRTLTextPlugin.isLoaded() || !this.hasRTLText) {
+                    text = transformText(formattedText, layer, feature);


Should there be a warning emitted here so that customers don't just see a map with no labels and wonder what the heck is going on?

I've changed the behavior now so that only when lazy loading, does it skip rendering the text.

…ired-plugin # Conflicts: # src/source/rtl_text_plugin.js

asheemmamoowala · 2019-10-17T17:23:27Z

src/source/rtl_text_plugin.js

+    unavailable: 'unavailable', // Not loaded
+    available: 'available', // Host url specified, but we havent started loading yet
+    loading: 'loading', // request in-flight
+    downloaded: 'downloaded', //plugin loaded and cached on main-thread and pluginBlobURL for worker is generated


What is the difference between downloaded and loaded, and why would we want to expose that distinction?

I generally don't think these statuses make intuitive sense. We shouldn't be asking end users to understand the intricacies of lazy loading the RTL plugin on the main thread and worker threads in order to get a status update. I hated unavailable when I included it but couldn't think of a better option. available meaning that the user has just specified a URL isn't intuitive. And I agree that downloaded probably shouldn't be exposed at all.

Removed, I've changed the architecture as described in the main PR body.

asheemmamoowala · 2019-10-17T17:25:07Z

src/ui/map.js

@@ -278,6 +278,7 @@ class Map extends Camera {
    _mapId: number;
    _localIdeographFontFamily: string;
    _requestManager: RequestManager;
+    _rtlTextPluginURL: ?string;


This is no longer needed right?

ryanhamley

I think overall this looks fine. It's a little tough to follow the code pathway but that's probably a natural consequence of having to pipe some of this information around the code base. My biggest concern is probably the various statuses (their naming and the states they represent).

Also, we need to make sure we have good documentation on the expected behavior of this plugin now.

ryanhamley · 2019-10-17T18:38:38Z

src/data/bucket/symbol_bucket.js

+
+                if (
+                    this.hasRTLText && globalRTLTextPlugin.isLoaded() || // Use the rtlText plugin shape text
+                    !this.hasRTLText || // non-rtl terxt so can proceed safely


nit: terxt

ryanhamley · 2019-10-17T18:38:53Z

src/data/bucket/symbol_bucket.js

+                if (
+                    this.hasRTLText && globalRTLTextPlugin.isLoaded() || // Use the rtlText plugin shape text
+                    !this.hasRTLText || // non-rtl terxt so can proceed safely
+                    !globalRTLTextPlugin.isAvailableInWorker() // We-doent intend to async-load the rtl text plugin, so proceed with incorrect shaping


nit: We-doent

ryanhamley · 2019-10-17T18:42:27Z

src/data/bucket/symbol_bucket.js

+
+                if (
+                    this.hasRTLText && globalRTLTextPlugin.isLoaded() || // Use the rtlText plugin shape text
+                    !this.hasRTLText || // non-rtl terxt so can proceed safely


This is minor, but it seems like the order here should be !this.hasRTLText || !globalRTLTextPlugin.isAvailableInWorker() || this.hasRTLText && globalRTLTextPlugin.isLoaded(). Start with the most generic check and work your way down to more complex ones.

ryanhamley · 2019-10-17T18:53:24Z

debug/rtl-plugin-autoload.html

+<script src='../dist/mapbox-gl-dev.js'></script>
+<script src='../debug/access_token_generated.js'></script>
+<script>
+mapboxgl.setRTLTextPlugin('https://api.mapbox.com/mapbox-gl-js/plugins/mapbox-gl-rtl-text/v0.2.3/mapbox-gl-rtl-text.js', null, true);


So the expectation here is that the plugin won't be loaded right away? How would we see that? Can this be modified to fly to somewhere with RTL text label names and emit an alert when the popup is loaded so we can see it working? And can this be combined with the existing rtl.html debug page?

I've made the plugin loading interactive in the rtl.html page, so you can click a button and see the status live.

ryanhamley · 2019-10-17T19:58:04Z

src/source/rtl_text_plugin.js

+    unavailable: 'unavailable', // Not loaded
+    available: 'available', // Host url specified, but we havent started loading yet
+    loading: 'loading', // request in-flight
+    downloaded: 'downloaded', //plugin loaded and cached on main-thread and pluginBlobURL for worker is generated


I generally don't think these statuses make intuitive sense. We shouldn't be asking end users to understand the intricacies of lazy loading the RTL plugin on the main thread and worker threads in order to get a status update. I hated unavailable when I included it but couldn't think of a better option. available meaning that the user has just specified a URL isn't intuitive. And I agree that downloaded probably shouldn't be exposed at all.

ryanhamley · 2019-10-17T21:04:20Z

src/source/rtl_text_plugin.js

    return callback;
 };

 export const clearRTLTextPlugin = function() {
    pluginStatus = status.unavailable;
    pluginURL = null;
+    pluginBlobURL = null;
+    lazy = null;


if lazy is a boolean, shouldn't it default to false?

ryanhamley · 2019-10-17T21:05:55Z

src/source/rtl_text_plugin.js

    pluginURL = browser.resolveURL(url);
+    lazy = !!lazyLoad;


Again, if we just default to false and always treated it as a boolean, we wouldn't have to do the cast here.

ryanhamley · 2019-10-17T22:04:04Z

src/source/rtl_text_plugin.js

+                throw error;
+            } else {
+                const rtlBlob = new window.Blob([data], {type: 'application/javascript'});
+                const URL = window.URL || window.webkitURL;


why do we need webkitURL here? i can't even really find much information on it but it seems to be deprecated

This may have been added in the past when Safari/Webkit didn't support window.URL. But that doesn't seem to be an issue any longer based on caniuse

src/util/script_detection.js

arindam1993 · 2019-10-17T22:47:53Z

I think overall this looks fine. It's a little tough to follow the code pathway but that's probably a natural consequence of having to pipe some of this information around the code base. My biggest concern is probably the various statuses (their naming and the states they represent).

Also, we need to make sure we have good documentation on the expected behavior of this plugin now.

I agree with this, I think I can change it to be a more uni-directional state-sync where the main-thread pushes the state of the plugin to the workers. The workers then just need to react to the correct state change and run importScripts when necessary.

I think that will let me get rid of the downloaded status as well.

- main thread now pings workers with an `pluginStateChange` event that syncs the state of the plugin on the main thread with the worker - worker logic is now reactive to this state change, and will load and parse the plugin on appropriate state change

asheemmamoowala

This is looking better. What is the performance impact of deferred loading during a flyTo animation?

Needs a docs/example update once merged (https://github.com/mapbox/mapbox-gl-js-docs/blob/publisher-production/docs/pages/example/mapbox-gl-rtl-text.html)

src/index.js

src/source/rtl_text_plugin.js

src/data/bucket/symbol_bucket.js

Co-Authored-By: Asheem Mamoowala <asheem.mamoowala@mapbox.com>

ryanhamley

I think the available status would make more sense if it were deferred instead and I don't think we need to check for webkitURL but otherwise, it looks pretty good

ryanhamley · 2019-10-24T18:50:13Z

src/source/rtl_text_plugin.js

-    unavailable: 'unavailable',
-    loading: 'loading',
+    unavailable: 'unavailable', // Not loaded
+    available: 'available', // The plugin URL has been specified, but loading has been deferred


maybe this status should be deferred. i still think available is confusing

ryanhamley · 2019-10-24T18:52:45Z

src/source/rtl_text_plugin.js

    return callback;
 };

 export const clearRTLTextPlugin = function() {
    pluginStatus = status.unavailable;
    pluginURL = null;
+    if (pluginBlobURL) {
+        const URL = window.URL || window.webkitURL;


think we can get rid of window.webkitURL unless you know of a specific reason to keep it

Autoload rtlTextPlugin using rtlTextPluginURL map option

f285c5f

arindam1993 requested a review from asheemmamoowala October 12, 2019 02:56

fix flow errors

7b55f59

arindam1993 added this to the release-sangria milestone Oct 14, 2019

asheemmamoowala reviewed Oct 14, 2019

View reviewed changes

Skip rendering rtl text if plugin is not loaded

ad15029

ryanhamley reviewed Oct 15, 2019

View reviewed changes

Merge branch 'master' of github.com:mapbox/mapbox-gl-js into rtl-requ…

5bb6ff6

…ired-plugin # Conflicts: # src/source/rtl_text_plugin.js

asheemmamoowala approved these changes Oct 15, 2019

View reviewed changes

arindam1993 changed the title ~~Load rtlTextPlugin on-demand using rtlTextPluginURL map option~~ [dnm]Load rtlTextPlugin on-demand using rtlTextPluginURL map option Oct 15, 2019

diferred laoding of plugin while not rendering text

51158d9

arindam1993 removed this from the release-sangria milestone Oct 16, 2019

asheemmamoowala requested a review from ryanhamley October 17, 2019 17:28

asheemmamoowala reviewed Oct 17, 2019

View reviewed changes

ryanhamley suggested changes Oct 17, 2019

View reviewed changes

Arindam Bose added 2 commits October 17, 2019 21:48

Fix lint issue

4fd3a66

arindam1993 changed the title ~~[dnm]Load rtlTextPlugin on-demand using rtlTextPluginURL map option~~ Load rtlTextPlugin on-demand using rtlTextPluginURL map option Oct 18, 2019

Remove rtlTextPlugin Map option

c3688f6

arindam1993 changed the title ~~Load rtlTextPlugin on-demand using rtlTextPluginURL map option~~ Add option to load rtl-text-plugin lazily Oct 18, 2019

Arindam Bose added 2 commits October 18, 2019 16:32

fancier rtl-text plugin demo page

ffe7dea

Lint

05db065

asheemmamoowala approved these changes Oct 21, 2019

View reviewed changes

src/index.js Outdated Show resolved Hide resolved

src/source/rtl_text_plugin.js Outdated Show resolved Hide resolved

src/data/bucket/symbol_bucket.js Outdated Show resolved Hide resolved

Arindam Bose and others added 2 commits October 22, 2019 17:06

Apply comments change suggestions from code review

89b3bcb

Co-Authored-By: Asheem Mamoowala <asheem.mamoowala@mapbox.com>

Switch to assert instead of throwing error

247226e

ryanhamley suggested changes Oct 24, 2019

View reviewed changes

arindam1993 self-assigned this Oct 24, 2019

Remove webkitURL and switch to using deferred

edc94ad

ryanhamley approved these changes Oct 24, 2019

View reviewed changes

Fix unit test

767c9dd

asheemmamoowala approved these changes Oct 24, 2019

View reviewed changes

arindam1993 merged commit d69e244 into master Oct 24, 2019

arindam1993 deleted the rtl-required-plugin branch October 24, 2019 23:23

arindam1993 mentioned this pull request Oct 24, 2019

Load RTL Text plugin on demand #8825

Closed

arindam1993 mentioned this pull request Dec 17, 2019

rtl-text-plugin fails to load if CSP for website doesn't allow script-src: blob: #9118

Closed

HarelM mentioned this pull request Mar 2, 2024

Overhaul of RTL plugin loading sequence maplibre/maplibre-gl-js#3728

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add option to load rtl-text-plugin lazily #8865

Add option to load rtl-text-plugin lazily #8865

arindam1993 commented Oct 12, 2019 •

edited

Loading

asheemmamoowala left a comment •

edited

Loading

arindam1993 commented Oct 15, 2019

ryanhamley Oct 15, 2019

arindam1993 Oct 18, 2019

asheemmamoowala Oct 17, 2019

ryanhamley Oct 17, 2019

arindam1993 Oct 18, 2019

asheemmamoowala Oct 17, 2019

ryanhamley left a comment

ryanhamley Oct 17, 2019

ryanhamley Oct 17, 2019

ryanhamley Oct 17, 2019

ryanhamley Oct 17, 2019

arindam1993 Oct 18, 2019

ryanhamley Oct 17, 2019

ryanhamley Oct 17, 2019

ryanhamley Oct 17, 2019

ryanhamley Oct 17, 2019

asheemmamoowala Oct 24, 2019

arindam1993 commented Oct 17, 2019

asheemmamoowala left a comment

ryanhamley left a comment

ryanhamley Oct 24, 2019

ryanhamley Oct 24, 2019

Add option to load rtl-text-plugin lazily #8865

Add option to load rtl-text-plugin lazily #8865

Conversation

arindam1993 commented Oct 12, 2019 • edited Loading

Launch Checklist

asheemmamoowala left a comment • edited Loading

Choose a reason for hiding this comment

arindam1993 commented Oct 15, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanhamley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arindam1993 commented Oct 17, 2019

asheemmamoowala left a comment

Choose a reason for hiding this comment

ryanhamley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arindam1993 commented Oct 12, 2019 •

edited

Loading

asheemmamoowala left a comment •

edited

Loading