Streamline build processes with esbuild

[ggetz]

Fixes #10373
Fixes #10400
Fixes #6464
Fixes #9906
Fixes #8245
Fixes #6993
Fixes #9388
Should fix #7669, verification needed
Workaround for #9416

How to build

For a single build, the workflow is the same as before:

npm install
npm run build
npm start

For an ongoing build task which incrementally rebuilds based on file changes, run:

npm install
npm run-build watch

and as a sperate process run:

npm start

For this to be truly a "just in time" build such that the build is run on npm start, we would want to run the equivalent of build-watch in server.js. In my mind at least, that would require a bit of a refactor of gulpfile.cjs so that the common build functions (a majority of the file) is either factor out into a separate file, or exported. Given we want to follow up this PR with revisions to the build scripts such as naming, I thinks it best to defer that change to a subsequent PR.

build now encompasses the functionality of both combine and build-specs, so they have been removed.

Ideally, for both simplicity and performance, we we use esbuild for workers as well, but we see the end result is significantly increased in size (1.1MB -> 10.1MB minified). I believe this is because there's no code splitting going on. However, code splitting is not supported for IIFE with esbuild, and Firefox still does not support ESM modules in workers. Potentially, we could output ESM with code splitting, and then post-process with rollup. This PR leaves building workers as is. When the Firefox support is there, it may make sense to holistically address our worker strategy.

Performance Stats

script	this branch	esbuild for workers	main
build (previously combine+ build-specs)	~7s	~4s	~25s + ~11s = ~36s
build-watch rebuild	300ms - 800ms	300ms - 800ms	~6.5s (and stuck in an infinite re-build loop)
build --minify --pragmas (minifyRelease)	~7s	~4.5s	~40s
buildApps	~2s	~2s	~30s

Size Stats

	this branch	main
minified	3.6 MB	4.3 MB
unminified	8.4MB	12.5 MB
source map	17.2 MB	22 MB

Highlights

• Bundles and minifies Cesium.jswith esbuild, resulting in faster build and build-watch times.
  • CI times have gone from 15 min -> 12 min. The bottleneck here isn't build/bundle time so much as tests and building documentation.
• Removes manual ThirdParty build/copy step; Instead use esbuild and resolve from node_modules directly
• There are no more AMD/UMD style modules; Instead the build output is IIFE for the browser, generic ESM modules, or CJS for node.
  • Other projects, like three.js, ship separate files for separate targets
  • CJS needs to be built differently in order to run (we use the "node" target esbuild), CC Regression: sampleTerrain no longer works in NodeJS #9906
• Due mostly to removing jsdoc comments, the unminified distribution is much smaller. We also get a slight benefit to the minified version as well.
• Make sure @legal comments are not removed during bundling or minification.
• Cut down on redundant build tasks; Instead we have fewer tasks with command line options (which shouldn't be needed as part of normal development). Also use a consistant kabab naming convention.
• Refine granularity of build-watch task, only rebuilding certain aspects depending on the file changed
• Remove convertToModules. This task was put in place for the ES6 module transition a few years ago, and we don't have a need to maintain it going forward.
• Test no longer use the Source code directly. Instead, they use the unminified IIFE build with source maps by default. Despite this, debugging still works as before since the debugger will handle the mapping back to the original source code. Therefor, there's no longer any need to have a build option or query anymore. The release option instead loads the minified IIFE build with pragmas removed instead, pretty much the same as before.
• Update build guide. Notably, minimum recommended Node version is now documented as 14. Adds engine field to package.json to warn about this.
• One big change in terms of ingesting Cesium is that, while you can still load the raw Source modules, we no longer build the workers to the Source/Workers directory. If a user want to do this, they should see the CESIUM_BASE_URL to one of the built directories. I'm not 100% sure this is the right decision.
• For coverage, we now instrument up-front using the Istanbul instrumenter directly, as opposed to using a karma plugin. This gives us more control, and allows us to build the code after instrumentation rather than instrumenting the bundled output (so we don't need a karma plugin anymore).

Known TODOS:

• Release packaging
  • Update makeZipFile task to reflect new files
  • Are the output files all named appropriately?
  • Should CSS files have a copyright header?
• Smooth out build task granularity, and update build guide
• Some workers are not loading correctly
• Specs are not rebuilding on watch
• Update testing guide
• Include GLSL build in build-watch task
• Can we add a quick spec to make sure Cesium is working in .cjs/node?
• Build warnings when running on travis? This worker code shouldn't be entered.

    > node_modules/@zip.js/zip.js/lib/core/codecs/codec-pool-worker.js:110:51: warning: "import.meta" is not available in the configured target environment ("es2015") and will be empty
  	return new Worker(new URL(workerData.scripts[0], import.meta.url), options);
  
  > node_modules/@zip.js/zip.js/lib/core/codecs/codec-pool-worker.js:110:51: warning: "import.meta" is not available in the configured target environment ("es2015") and will be empty
  	return new Worker(new URL(workerData.scripts[0], import.meta.url), options);
  

• Build Sandcastle
• Update remove RequireJS from credited third party libraries
• coverage shouldn't include third party modules
• Make sure third party licensing is not removed during minification (Keep third-party code license under minification? #9388)
• Do an optimization pass; esbuild is fast, but biggest bottlenecks will be our JS plugins
• The workers have significantly increased in size (1.1MB -> 10.1MB minified). I believe this is because there's no code splitting going on. However, code splitting is not supported for IIFE with esbuild, and Firefox still does not support ESM modules in workers. Potentially, we could output ESM with code splitting, and then post-process with rollup. Or we can rollback to rollup for web workers entirely for the time being until the Firefox support is there, at which time it may make sense to holistically address our worker strategy.
• Update CHANGES.md

[cesium-concierge]

Thanks for the pull request @ggetz!

• ✔️ Signed CLA found.
• ❔ CHANGES.md was not updated.
  • If this change updates the public API in any way, please add a bullet point to CHANGES.md.
• ❔ Changes to third party files were made.
  • Looks like a file in one of our ThirdParty folders (ThirdParty/, Source/ThirdParty/) has been added or modified. Please verify that it has a section in LICENSE.md and that its license information is up to date with this new version.

Reviewers, don't forget to make sure that:

• Cesium Viewer works.
• Works in 2D/CV.

[ggetz]

@sanjeetsuhag I'm still making a few tweaks based on the final TODO's, but most of this PR should be static. Could you give this a try and let me know if you have any general feedback?

[sanjeetsuhag]

@ggetz It's amazing how much faster all the scripts are working. Can't wait to see how much of an impact this will have on CI. The debugging workflow for workers is much easier now too! Otherwise, I just had a few comments.

[ggetz]

Thanks @sanjeetsuhag! Your feedback should all be addressed.

[ggetz]

@sanjeetsuhag Could you please give the latest updates a round of review? Assuming this is in a good place, it would help to get this merged and move #10538 along as well.

[sanjeetsuhag]

@ggetz This is very close. I just had some minor comments. One big thing in running the code - it seems like the build step is slow.

On this branch, I ran git clean -xdf; npm install; npm run build and the build step took 5.84s. I switched to main and ran the same command and the build step took 5.04s.

[ggetz]

One big thing in running the code - it seems like the build step is slow. On this branch, I ran git clean -xdf; npm install; npm run build and the build step took 5.84s.

This is mainly due to reverting back to rollup to build the workers for the reasons detailed in the PR description. The up-front build here does more than it does in main - Including bundling third-party code, building specs, and copying static files. In main the totality of these tasks was 36s. The iterative builds are still much faster and will be run by default in #10538.

[ggetz]

Thanks @sanjeetsuhag! This is updated.

[sanjeetsuhag]

Everything seems to be in good shape. This is an awesome improvement! Thanks @ggetz! We should also update the Release Guide since it references outdated steps like build-specs.

[mramato]

This should probably be absolutely top of the list with 📣, it's not a breaking change and it's a huge deal.

[mramato]

Doesn't Cesium now 100% require a bundler to use modules? (And a bundler that allows for node resolution rules at that). Or are we still embedding third party code?

[ggetz]

We're embedding the third-party code in these builds. In the blog post I mention that production apps that use their own bundler should be using modules directly Source, which does not have third party code bundled. Which is what we've previously recommended as well if I understand correctly.

[mramato]

We have previously recommended it, but since we embedded ThirdParty code in source, you didn't need a bundler to use them. Now you do.

Another problem I just noticed is that package.json lists everything as a dev dependency, which may not be correct because that means if I npm install Cesium and then try to use Source it's going to pull in modules that don't exist because they are not a dependency or peerDependency.

[ggetz]

Gotcha, I see the distinction. For the point about the bundler, I will update both CHANGES.md and the blog post with the requirement.

For the latter, I will package.json such that modules in Source/ThirdParty are now marked as direct dependencies. Sound correct?

[mramato]

Yes. It's unfortunate because people who want to use the bundled Cesium.js still need to sync those dependencies but there's no other way that I know of to do it and have all usage-types work.

[thw0rted]

Hey all, speaking of package.json, would this be a good time to address #9212 as well? It's really important to get the exports right, I had a workaround but had to get rid of it to be able to use another important dependency at all...

ETA: to clarify, any resource that you intend for people to be able to bundle or extract separately (CSS, images, maybe data files like terrain height / IAU JSON) should have its own exports entry, and excluding those entries makes consumption of the resources effectively impossible.

[ggetz]

Hi @thw0rted, thanks for pointing this out! We have some additional thoughts on related improvements beyond just this PR, including considering #9212. We'll add our thoughts or any updates to that issue.

[ashley-mort]

Does this or will this provide any mechanism to preload the workers up front? Loading ~50 web worker files on our network kills us by delaying user polygon drawings.

[thw0rted]

I was going to say that your application might be able to improve the situation using link rel="preload" tags, but at least in Chrome, maybe not. Might help in other browsers though? I have not benchmarked, but would be interested in learning what provides the best user experience..

[ggetz]

@ashley-mort If you are using the Primitive API, there's a asynchronous option which you can set to false to draw geometry synchronously instead. That may help with your use case.

We're also exploring optionally inlining static assets in a single distributable, which would include the web workers with the initial CesiumJS download.

[ashley-mort]

The inlining static assets in a single distributable seems like it would be great! I have tried out the c137.js stuff and it was a huge improvement regarding the time it takes to display user created polygons on the network we have.