Switch to ES modules and Rollup

[anandthakker]

Closes #5939
Refs #6056

This PR:

• Refactors the codebase to use ES modules
• Changes the bundler from browserify to Rollup

Note on building the Worker bundle blob:
There's a substantial change here in our approach to constructing the Blob URL for the worker script, because webworkify depends on specific browserify implementation details. The way we're doing it now:

• Use rollup's "code splitting" feature to run the build against two entrypoints: src/index.js and src/source/worker.js. The result is three bundles: rollup/build/index.js, rollup/build/worker.js, and rollup/build/chunk1.js. The first two correspond to the two entrypoints, and the third consists of all the code that is shared by the main thread and worker scripts.
• Use a second pass to assemble these three chunks into a single bundle, with some webworkify-inspired wrapper code that, at runtime, stringifies the worker source and uses it to create our worker blob URL.

Note on running our code in Node:
In order to run our Flow-annotated ES module code in a Node environment, we have to use two require hooks. Previously, we would do this for flow by doing require('flow-remove-types/register') in our common unit test module -- this allowed us to require flow-annotated files from tests, but did not permit flow in the test files themselves. Instead, we're now telling node to require flow-remove-types (and @std/esm, for modules) directly via command-line arguments; you can run node or tap with these args already set using the build/run-{tap,node} scripts.

Benchmarks

Full
Re-run of Paint

Time to first render:

rollup-anandthakker:

master

Minified bundle size:

-rw-r--r--  1 anand  staff   709K Mar  7 15:37 dist/mapbox-gl--master.js
-rw-r--r--  1 anand  staff   572K Mar  7 15:42 dist/mapbox-gl--rollup-anandthakker.js

-rw-r--r--  1 anand  staff   169K Mar  7 15:37 dist/mapbox-gl--master.js.gz
-rw-r--r--  1 anand  staff   148K Mar  7 15:42 dist/mapbox-gl-rollup-anandthakker.js.gz

Todo

• Check whether "minifying" the style-spec (as in https://github.com/mapbox/mapbox-gl-js/blob/master/build/minifyify_style_spec.js) eliminates the perf regression in the worker-gets-the-whole-bundle approach NOPE.
• Get code splitting to work
• Polish build
  • Prod / dev variations
  • Ensure fast watch/rebuild cycle (upstream PR submitted: Enable watcher in code-splitting mode rollup/rollup#2035)
  • Sourcemaps 😨
  • style-spec package build
  • benchmarks build
  • npm scripts / CI
  • debug pages build (mainly access_token_generated.js)
• Determine the best way to execute es module code in Node environment and apply it to:
  • Unit tests
  • Integration tests
  • Codegen scripts
• Update docs site
• Update lint rules
  • Use https://github.com/benmosher/eslint-plugin-import h/t @mourner
• Remove single module object export pattern (const exported = {...}; export default exported;) -- ticketed Replace exported objects with named exports #6293
• Remove browserify, (+ unused browserify transforms, derequire)
• Post benchmark results: Full / Re-run of Paint
• Post final time-to-first-render comparison

[mourner]

We should probably get rid of all instances of exporting object literals like this — I suppose this is a legacy require/codemod thing? Ideally we would always export symbols directly, and only export default for classes/functions. This makes tree-shaking more effective.

[anandthakker]

👍 yeah these cases were either codemod outputs (or just the quickest manual fix for the time being). I'm not sure how much tree-shaking we'll enjoy from within our codebase, since I don't think we have too much dead code within src/... but if we do end up going the code-splitting route, this could make a notable difference in the worker blob size. It'll also just be simpler/clearer code.

[jfirebaugh]

Can you ticket this as a followup item?

[anandthakker]

Yep, already did: #6293

[mourner]

Committed accidentally?

[mourner]

Should we remove those benchmarks as a part of this PR?

[anandthakker]

We can reinstate them or remove them in a separate PR -- I was just getting distracted by the error messages coming from these files :)

[mourner]

Can you expand on why flow-remove-types and @std/esm don't play well together? It would be great if we could avoid generating the source to a separate src_untyped folder...

[anandthakker]

Some context here: standard-things/esm#119

[anandthakker]

It would be great if we could avoid generating the source to a separate src_untyped folder...

I agree, I'm really hoping we can avoid a build step for tests 🤞

[mourner]

BTW how's the total bundle size (minified/gzipped) looking?

[anandthakker]

@mourner pretty good!

master

-rw-r--r--   1 anand  staff   682K Dec 12 23:23 mapbox-gl.js
-rw-r--r--   1 anand  staff   168K Dec 12 23:23 mapbox-gl.js.gz

rollup-anandthakker

-rw-r--r--   1 anand  staff   555K Feb 22 11:13 mapbox-gl.js
-rw-r--r--   1 anand  staff   144K Feb 22 11:13 mapbox-gl.js.gz

[anandthakker]

Alright, using Rollup's relatively new code splitting feature, we're now at a point where we have a smaller bundle that loads as fast as our browserify build. Before I embark on the longish tail of tasks that would be needed to finish this and merge it into master, it's time to ask: should I?

The advantages:

• A smaller bundle, with the potential for (possibly significant?) further optimizations via "tree shaking"
• Interesting build possibilities: with rollup & es modules, we could experiment with a build of GL JS that uses dynamic imports to fetch modules on demand.
• Modernized codebase:
  • language-level module declarations (rather than runtime ones via require()/module.exports)
  • better handling of cyclic dependencies
  • cleaner code (IMO, anyway)
  • consistent with our ubiquitous Flow import type / export type declarations

The disadvantages:

• Browserify has been around a long time, is well hardened, we know it works.
• Requires the use of Rollup's new "code splitting" functionality, which is still new and behind a feature flag (experimentalCodeSplitting). (On the other hand, there are a couple of active core maintainers who have already been very responsive.)
• Might require something like a build step to handle the combination of ES modules + Flow before we can run code in Node (for unit tests, integration tests, codegen)

@mapbox/gl-core what do you think? Please weigh in! It's definitely not too late to abandon this effort if we don't feel that the time is right.

[ChrisLoer]

I can't really comment on the overall approach although it sounds like progress! Two questions:

• How would "dynamic imports to fetch modules on demand" work? I'm wondering if we'd end up building out infrastructure that would make it convenient to have mapbox-gl-rtl-text auto-load instead of requiring additional configuration. Even closer to the dream would be to dynamically choose whether to load a WASM or ASM.js version of the plugin based on browser capability.
• You've been doing a lot of bechmarking of time-to-first-render, right? Can you document any of those numbers here?

[jfirebaugh]

Might require something like a build step to handle the combination of ES modules + Flow before we can run code in Node (for unit tests, integration tests, codegen)

I think this is the most important unanswered question. Way back in the day we reverted an early attempt to introduce ES6 syntax because tooling wasn't ready -- specifically debuggers and code coverage. I rely on running individual test files pretty frequently, often attaching a node debugger to track down test failures. If switching to ES6 modules would mean I can no longer do that, I'd prefer to hold off on it.

[anandthakker]

🎉 looks like flow-remove-types and @std/esm can cooperate after all! @jfirebaugh I confirmed that I'm able to attach a debugger with node --inspect and step through original sources.

[anandthakker]

Another open question: how should we build the benchmarks? Options:

1. Reproduce the code-splitting trick we're using for the main bundle. Pro: we continue to have a single benchmarks.js bundle that gets published on each master CI build and for each release. Con: more complicated build setup.
2. Generate a separate worker.js script, and, in benchmarks.js, set mapboxgl.workerUrl to point to it. This would mean that the web workers would be loaded from that worker.js file rather than a blob URL. Pro: less complicated build setup. Con: we would have to publish two files--benchmarks.js and worker.js.

I was initially thinking that another downside to option 2 is that it would preclude being able to benchmark anything that included worker load time, because we'd be loading the worker in a different manner than in the shipped library. But in fact, I don't think it would really be practical to measure this within our benchmarking system regardless of how we load the workers, so I don't think this should be a factor.

[anandthakker]

How would "dynamic imports to fetch modules on demand" work? I'm wondering if we'd end up building out infrastructure that would make it convenient to have mapbox-gl-rtl-text auto-load instead of requiring additional configuration. Even closer to the dream would be to dynamically choose whether to load a WASM or ASM.js version of the plugin based on browser capability.

@ChrisLoer I haven't dug into the details here, but I think dynamic imports might indeed make this a possibility. (But we'd likely have to also have to keep supporting the existing approach for the foreseeable future.)

You've been doing a lot of bechmarking of time-to-first-render, right? Can you document any of those numbers here?

Not much more than what's documented in #5939 -- I'll do another pass at this again soon, along with running our normal benchmarks, and post results here.

[jfirebaugh]

Revert this to fix the build (zaach/jsonlint#103 (comment)).

[mourner]

Generally I think this is great, and we should move forward. It's a bit scary for sure, but ultimately feels like the right choice for future.

Apart from the remaining bullet points and making the build green, I'd love to add a bunch of eslint-plugin-import rules to our ESLint config — used this plugin and it is fantastic. Can help catch a lot of subtle bugs.

Also, would be great to fix a few remaining export const exported = { ... } cases, but I'm fine with leaving it for a follow-up PR if it makes it easier to ship.