Make ES6 (ESM) and ES5 (CommonJS) versions, and make ES6 the default.

[dpvc]

This is a substantial update to the original ES6-module PR that was the original PR.

Don't panic about the number of files!

More than 300 of them are due to renaming of files or combining several json files into a single new one. There are over 130 deletions in components/src due to the json file overhaul that combines the old build.json, copy.json, and webpack.config.js files into a single config.json file that contains the data needed for all three. That also means that there are 73 new config.json files that are just that new combined data that you should not have to look at. There are also about that same number of component .js files that have had a path name change and nothing else. Finally, the legacy AsciiMath files needed to be moved in order to get the v2 CommonJS files separated from the new ES-module files that glue the v2 files into v3. That is about 25 more changes that are just moved files. These amount to over 300 of the 380 file changed in this PR. If you close the components/mjs, components/src and ts/asciimath/legacy directories in the sidebar, you will see that there are really only about 60 actual changed files. Many of those are just changes in file paths (see below for more information about path changes). The only really substantive changes are in the build tools in components/bin, the webpack files in components, the scripts in the package.json file, and separating out some bits that have to change between CommonJS and ES-modules (e.g., access to __dirname) into separate files that can be included in the CJS and MJS versions as needed. There are some changes in hooking in the v2 AsciiMath code as well. But that is basically it; the rest is really just restructuring the directory layout and component control files.

New directory organization

The main change in this version is to make both CommonJS and ES-module versions of the MathJax code, both from the same original typescript files. In the past, the typescript files were compiled as CommonJS into the js directory, but now we have two directories, cjs and mjs for the two forms, which are compiled using separate tsconfig.json files. The latter are stored in the tsconfig directory as tsconfig/cjs.json and tsconfig/mjs.json. The top-level tsconfig.json simply references one of these two (the mjs version by default). The cjs.json and mjs.json files both reference common.json, which includes all the common settings for the two, and the cjs and mjs files set the module-specific values (the target, outDir, etc.). These also include a paths array, which is what allows us to select the module-specific versions of the typescript files. This is accomplished by using a pseudo-module, such as #root, that can be redirected to either ts/components/cjs or ts/components/mjs depending on which version is being compiled, thus providing support for both module formats.

The compiled .js files retain the reference to #root, however, and so we use the imports property of the package.json file to resolve these to the proper location as well. The main package.json file, which sets "type": "module" indicating that the '.js' files any subdirectories are ES modules, also sets #root, this time to mjs/components/mjs/, so that node (and webpack) will redirect #root to the mjs-specific root.js file. For the cjs directory, after compilation is completed, the script that does the compiling also creates a package.json file in the cjs directory that includes "type": "commonjs" and an imports property that makes #root to cjs/components/cjs/root.js, overriding the main package.json package to mark the files in the cjs directory as CommonJS modules, and pointing #root to cjs/componts/cjs so that when node or webpack loads files from the cjs directory, #root/root.js will be the cjs version rather than the mjs one.

Similar techniques are used to handle the node modules used by MathJax. For example, the #menu pseudo-module is redirected either to mj-context-menu/mjs or mj-context-menu/js, as needed, and similarly for SRE, mhchemparser, and the default font. Finally, there are #js and #source pseudo-modules used in the components control files to access either the cjs or mjs directories, and for the source.js file to get access to the __dirname value properly.

In the past, the webpacked files were stored in the es5 directory. That reflected the fact that they were compiled with target: ES5, but as we are chaining to ES6 as the default, that would necessitate a change to es6 or some other name. We have chosen bundle so that it doesn't need to change in the future if we update again. The webpacked files can be considered CommonJS file so they can be included into either mjs- or cjs-based node applications, so the bundle directory gets a package.json file that sets the type to commonjs in order to make that possible.

Because the components control files use import and export, they are ES modules, and so component/src has been renamed components/mjs. In order to support commonjs programs that load the components from source, there is a package script that down-compiles the components/mjs files to a new directory components/cjs (using tsconfig/components.json), and the resulting directory gets a package.json like the one in cjs that makes them as commonjs modules and redirects the pseudo-modules to the cjs versions. In particular, any references to #js will go the main cjs directory rather than the mjs directory. This allows the component source to be used in either type of module.

In order to support by module types, the node-main component needs special handling. There is now separate entry points for cjs and mjs modules. These first set up the __dirname and require values using the techniques needed for the module type, then loads the shared node-main.js file that does the main work. The small module-specific bootstrap files can't be webpacked (webpack doesn't have the module library needed for the mjs version, and if the cjs version is packed, it will record the directory as being the original one in the components directory, not the webpacked file's current location in the bundle directory), so only the node-main.js file is packed, and the cjs and mjs bootstrap files are copied to the bundle directory along with the webpacked file. That means they will do their usual setup, and then load the packed node-main.js file. This makes it possible to include node-main in the slimmed down mathjax npm package that only includes the webpacked versions, so that those who use npm install mathjax@4 will be able to do require("mathjax").init({...}) as in version 3, or import {init} from 'mathjax' just as they can for the mathjax-full package.

This ability to require or import mathjax and mathjax-full is made possible by including an exports section in the main package.json file. This includes

    ".": {
      "import": "./bundle/node-main.mjs",
      "require": "./bundle/node-main.cjs"
    },

which is what allows require("mathjax-full") to load the cjs bootstrap file, while import {init} from 'mathjax-full' loads the mjs one. Similarly, we include

    "./source": {
      "import": "./components/mjs/node-main/node-main.mjs",
      "require": "./components/cjs/node-main/node-main.cjs"
    },

which lets require("mathjax-full/source") and import {init} from 'mathjax-full/source' load node-main from the source code (so you can test out changes without having to repack everything).

The package.json includes other such mappings that are designed for backward compatibility:

    "./js/*": {
      "import": "./mjs/*",
      "require": "./cjs/*"
    },
    "./components/src/*": {
      "import": "./components/mjs/*",
      "require": "./components/cjs/*"
    },
    "./es5/*": "./bundle/*",

Since v3 used a single js directory that is now two directory, cjs and mjs in v4, the first mapping above redirects require('mathjax-full/js/<file>') and import ... from 'mathjax-full/js/<file>' to be the new cjs and mjs directories (without the need for symbolic links). Similar for components/src which now has cjs and mjs versions. The es5 directory is rerouted to the new bundle directory. Finally, the mapping

    "./*": "./*"

allows any other require or import to go through normally. Since exports restricts the files that can be included into an external node application to those in the exports section, if there is one, this allows direct imports of any MathJax mjs, cjs, or bundle file into a node application (so this supports all the techniques described in the MathJax-demos-node repository).

The new components control files

The files that control the component build process have been significantly updated. In the past, there were separate build.json, copy.json and webpack.config.js files in each component/src directory that needed to perform one of those functions. This PR consolidates the data from those three files into a single config.json file for each component. The JSON data in this file has sections for building, copying, and webpacking, with sections being present if the component needs to perform that function.

The build and copy sections of config.json are essentially just the original data from the build.json and copy.json files. The webpack section comes from the data that used to be passed to the PACKAGE() function in the old webpack.config.js files. This data is read by the updated components/webpack.common.js file and it calls the PACKAGE() command itself using that data. This simplifies the data needed for each component, and consolidates all the webpack configuration creation into the main webpack file. (The new webpack.common.js file does look for a webpack.js file in the component directory, and if present, it should return a function that accepts a webpack configuration as its argument and returns a possibly modified configuration that is the one that will actually be used during the webpacking. This allows a component to modify the webpack configuration in situations where the default one is not sufficient, though this feature is not currently used in MathJax or its fonts.)

The new webpack data has defaults for values that used to be required to be passed to PACKAGE(), so the webpack data is smaller for the components than in the past. That is, the location of the MathJax js files and the current directory no longer need to be included, and there are a few additional values that can be included, making some of the PACAKGE() post-processing that used to be done in the webpack.confo.js file unnecessary.

Although we only webpack the mjs files for v4, I originally was webpacking both the mjs and the cjs versions, and the infrascture is still there in case anyone needs ES5 versions of the packed files (created from the cjs javascript rather than the mjs ones). This is controlled by having two webpack.config files in the components directory, webpack.conf.mjs and webpack.conf.cjs. The former packs the ES6-based files from the mjs directory into the bundle directory, whereas the latter packs the ES5 files from the the cjs directory into bundle-cjs (which is not normally needed, but the tools are there to make it if needed). The two module-specific config files each sets things up for the proper module type, loads PACAKGE from the webpack.common.cjs file, which does most of the work, and uses it to create the proper webpack configuration from the component's config.json file.

The components/bin/pack command calls webpack using a command like

npx webpack --env dir=components/mjs/input/tex -c components/webpack.config.mjs

or

npx webpack --env dir=components/cjs/input/tex --env bundle=bundle-cjs -c components/webpack.config.cjs

and the webpack.config file gets the directory to pack from the first --env value and the alternative bundle directory to use from the second (if any).

Note that since the components/cjs javascript files are down-compiled to ES5 CommonJS modules, there is no longer a need for the babel translation that used to be done on the files in components/src, so the webpacking is quicker and relies on fewer dependencies.

The handling of the libs values in webpack.commom.cjs has changed: in the past, a single regular expression was used to determine if a .js file needed to be redirected to a lib file that would get its values from the MathJax._ variable, whether it came from the js directory in the PACKAGE() configuration or from the MathJax js directory. For fonts and user-defined custom code that uses the js option to specify code outside of MathJax-src, this could be a problem, since a file with the same name as one in MathJax-src/mjs could incorrectly be redirected to a MathJax component lib file (this occurred when I changed the structure of the font directories, and is what necessitated the extra directory in the font ts and js directories in an earlier PR). Now, separate regular expressions are used for each entry in the libs array, keeping the js and components lib files separate, so that there is no accidentally overlap (and the extra directories for the fonts can be removed again, making that structure cleaner). Finally, the custom plugin used to make the library substitutions now uses a hook into an earlier part of the resolution process: in the past, we hooked into the file and module events, but now use the normalResolve hook, and use doResolve() with a modified copy of the request object rather than chainging the original request, which apparently is the sanctioned way of doing this.

The build tools in the components/bin directory are modified to handle the new mjs and cjs directories, the new bundle directory, the new config.json files, and the new webpack.config files. The makeAll command now has more command-line options that tell it which format to build (mjs or cjs), which bundle to use, which actions to take (build, copy, pack), and how verbose to be about the output. (In the past, the package scripts used grep to filter the output, which Windows users didn't have. I've tried to make the package scripts be less dependent on a unix environment, and so they use node programs to do most of the work, even if easier unix commands might be available.) The components/bin directory also includes a package.json file to make these programs be commonjs modules (so they don't need the extensions that would be required by ES modules). This package.json is also the one that is copied to the cjs and components.cjs directories when they are created, so it includes the imports mappings needs by those directories, even though they are not needed for components/bin.

Package scripts

Because there are now two versions (cjs and mjs) to be created, and two forms of the components, the scripts for compiling and building MathJax are more complicated, and there are lots of new scripts, many just for internal use (these are usually marked by an initial underscore in their names). Some come in two forms, one for cjs and one for mjs, and a third, which defaults to the mjs form. For example, you can do npm run -s compile-mjs, npm run -s compile-cjs, or just npm run -s compile (which defaults to npm run -s compile-mjs). There are new command for down-compiling the components source to components/cjs, a new npm run -s make-one <component> <mjs/cjs> that makes packing a single component easier, and new build scripts npm run -s build, npm run -s build-css and npm run-s build-all to do both the compile and make-components functions all at once, for one or both module types.

Because some files refer to mathjax-full/..., there is an install script that makes a symbolic link in node_modules to the MathJax-src directory, when MathJax-src is obtained from git rather that as an npm package, and npm install is run, so that those references will work properly.

Finally, the tsconfig.json file calls on the tsconfig/mjs.json file so that the emacs typescript mode will be able to find the configuration properly and compile into the mjs directory as you edit files. If you need to work on the commonjs in cjs instead, you can use npm run -s use-cjs to switch tsconfig.json to call tsconfig/cjs.def so that emacs and nix tsc will compile into that directory. Of course, npm run -s use-mis will put things back to normal.

The v4 lab

One of the advantages of using ES-modules rather than CommonJS is that these can be used directly in the browser via <script type="module"> without the need for System.js or the down-compiling done by traceur.min.js. I have a modified version of the v3-lab.html for use with the mjs files, and will make a PR in the MathJax-dev directory for that, so that you can try this out and test this PR. The new lab uses <script type="imagemap"> to redirect the pseudo-modules to their proper locations (just like the import section of the package.json file, and the paths array of the tsconfig files). With these redirections, the mjs files load into the browser directly.

Fonts with ES modules

Just as MathJax now has both cjs and mjs forms for its javascript files, the MathJax fonts need to have both versions (and in particular, the mjs versions need to be used for the v4-lab). That means the font packages need to be updated as well. I have these ready, and will publish v4-beta versions of these that can be used to test this PR. When I do that, I will make another commit that updates the versions in the package.json file.

Node module dependencies

Just as with the font files, the node modules on which MathJax depends need to have both mjs and cjs versions (webpacking could get away with only cjs versions, but to use the mjs versions directly in the browser, we need mjs versions of the dependencies as well). Thes are now available for mj-context-menu, speech-rule-engine, and mhchemparser, as well as the MathJax fonts. In order to be able to switch which version is being used, we need to use pseudo-modules, and redirect them via the package.json, tsconfig.json and <script type="image map"> files (web pages that load webpacked versions don't ned to worry about this, only the lab).

To make this easier, there is a new ts/ui/menu/mj-context-menu.tsfile that works likets/a11y/sre.tsto do all the importing from mj-context-menu, and exports the needed pieces. Thus all the MathJax files that need anything from mj-context-menu will import them from../ui/menu/mj-context-menu.js, and only that one file needs to use the #menupseudo-module. Similarly,ts/a11y/sre.tsimports from#sre, and all other MathJax files import from ../a11y/sre.jswhat they need. So a number of files have changes just in the imports to go from the mj-context-menu node module to../ui/menu/mj-context-menu.js` instead. This also allows multiple imports to be collapsed into one, since they all now come from the same file.

Similarly, there is an #mhchem pseudo-module for the mhchemparser node module, and #default-font for the mathjax-modern-font node module. This would make it easier to change the default font if it should change in the future: one merely changes the package.json, tsconfig.json and <script type="imagemap"> remappings, and no MathJax code has to be changed.

To cover all the different usage cases (webpacked versions in browsers, ES source in browsers, webpacked versions in node apps, component source (cjs and mjs) in node apps, and direct module access (cjs and mjs) in node apps), it turns out that using export default made some things more difficult, so this PR switches from import Sre from '../a11y/sre.js to import {Sre} from '../a11y/sre.js, and a few similar cases.

The global object

Covering all the cjs/mjs/browser/node combinations meant that the way the global object was handled in ts/components/global.js needed to be modified to cover more cases. The changes are isolated to that file, which is used to obtain the global MathJax object, and handle its configuration values.

AsciiMath

Because the AsciiMath files are still the v2 versions, they are CommonJS, and so need to be marked as such via a package.json file. The '.ts' glue files that attach the v2 files to the v3/v4 ones needed to be separated from the v2 files so that they could be treated as ES modules in the mjs directory and CommonJS cjs. That means that some reorganization was need in the ts/input/asciimath directories to make that separation work. This required moving some code that had been added to ts/input/asciimath/mathjax2/input/AsciiMath.js to be moved out to new ts/input/asciimath/legacy.ts and ts/input/asciimath/shim.ts files, and moving ts/input/asciimath/mathjax2/legacy up one directory to ts/input/asciimath/legacy (no need for the extra mathjax2 directory).

[dpvc]

With the new ES6 modules generated by this PR, it is possible to use these directly in the development lab via <script type="module" src="...">, without the need for system.js. But in order to do that, we need the node dependencies (mhchem, mj-context-menu, and speech-rule-engine) to include ES6 modules as well. Currently, they are CommonJS modules, which can't be used via <script type="module">, but if I recompile into ES6 modules (and include the proper import map), it all works nicely.

So it might be worth considering making es6 the default and having js be the es6 versions, with js5 rather than js6 being the additional version. This would be another potential breaking change for node users, since ES6 modules and CommonJS can't be mixed (without -r esm), but this might be the right time to make that change.

What do you think?

[dpvc]

PS, note that the webpacked versions of the components seem to be OK with the mix of ES6 and CommonJS, so we are able to get away with the current versions of the node libraries as they are now, but for direct inclusion into node apps or <script type="module">, we need the actual ES6 versions.

[pkra]

So it might be worth considering making es6 the default and having js be the es6 versions, with js5 rather than js6 being the additional version. This would be another potential breaking change for node users, since ES6 modules and CommonJS can't be mixed (without -r esm), but this might be the right time to make that change.

+1 from this bystander.

[dpvc]

+1 from this bystander.

Thanks. I had already counted you in on that. :-)

[dpvc]

@zorkow, hold off on this one for now. I've been working on making ES6 the default, and am having a number of problems with that. We will need to discuss some things own our next meeting, but I'm slowly making progress. There are some important changes needed in the components directory and I'm not done with it yet.

[dpvc]

I have updated this PR to the new ES6-module code. I have replaced the description at the top to now describe the current PR contents, which are a substantial update from the earlier draft version.

Don't panic about the file count! See the section on that at the top of the description, as it show how most of the file changes are due to renaming files, and combining several json files into one for the control files for the components. So read that section before proceeding!

[zorkow]

A long read... and not many comment.
I believe the most important part is that the explanation of the PR makes it into the MathJax documentation somewhere!

[zorkow]

So we no longer build with default locales?

[dpvc]

As you note below, this util-lab.js file is only used by the v4 lab. The reason is that node, webpack, and the browser each need the json files to be handled differently. Webpack requires the assert { type: 'json' } hint, but node won't use that without an experimental command line argument, while the browser won't import json at all. So I needed to make a separate util file for each case. This is the browser one, which doesn't pre-load the locales, but will use its own loading mechanism to do so when needed. The util-pack.js file uses includes the the assert hints, while the util.js file used by node when running node-main with component source files uses require to avoid the assert issue with import. This works in both mjs and cjs settings since our node-main.mjs sets up require so that SRE will be able to load files, so we take advantage of that here.

[zorkow]

I see, that is here now.

[dpvc]

This is the file used by webpack.

I suppose I could have isolated just these four lines plus the setting of the maps into a second utility file and used three versions of that instead. I can do that if you prefer.

[zorkow]

Does the -lab indicate this is for the dev lab?

[dpvc]

Yes. There is another file that does that as well (source-lab.js).