Endo

[kriskowal]

This tracks the full stack of changes needed to build a new core for importBundle / importArchive. The new endo package implements importPath, writeArchive, and importArchive, such that a round-trip through an archive behaves identically to executing directly.

The endo name implies an aspiration to provide a drop-in replacement for node, but encapsulated. This cut doesn’t engage in that aspiration, other than to lay groundwork. There is far more TODO than TODONE.

The stack, at time of creating this draft, also includes a straw man archival format, which will be replaced with a suitable implementation of zip or tar. Many deep packet inspection firewalls filter zip files. Tarballs have a limit of 100 bytes or 255 bytes for each entry’s path, depending on the sophistication of the implementation. That limit could be overcome by using hashes and a detailed manifest, but that would defeat the ideal that an extracted archive be just as executable.

The strawman archive library (agar) can run in SES, doesn’t take a dependency on fs or anything else Node.js-specific, so is suitable for front-end as well as back-end. The format has no limitation on file name length or file size and doesn’t waste space or complexity on Unix or Windows-specific features. It is, nonetheless, temporary.

Regardless, the archival format should be loosely coupled. The Endo archive builder expects an object that implements {get, set} or a function that creates such an object from a byte array.

• module specifier and url math feat(endo): Module specifier and URL math #342
• infer exports feat(endo): Infer the module map a package exports #350
• map compartments feat(endo): Add compartmap #344
• search feat(endo): Add search #353
• assemble feat(endo): Add assemble #345
• import feat(endo): Add import #351
• zip feat(endo): Add zip #346
• ses compartment load feat(ses): Add Compartment load function #349
• archive feat(endo): Add archive #347
• import archive feat(endo): Add import-archive #352
• main feat(endo): Interface tests #348
• CJS and JSON feat(endo): Add CommonJS and JSON module support #397
• support Node.js index.js conventions
  • import index.js feat(endo): Search for .js or /.index.js #401
  • SES support for importHook redirects or aliases feat(ses): Support importHook alias returns #432
  • archive index.js feat(endo): Search for index.js and implied package exports #434
  • consolidate import hooks (refactor) refactor(endo): Consolidate archive and import hooks #436
  • fix referrer specifier fix(endo): Fix index.js redirects #437
• type=module feat(endo): Infer module type from package.json "module" #402
• minimum viable command-line feat(endo): Add command line #398
• parsers override feat(endo): Support package.json parsers override #404
• CLI support for subverting node in a subshell feat(endo): Support exec shell environment #411
• cross-package linkage in absence of “main” and “exports” (where anything not masked by “files” is eligible to be imported across packages. feat(endo): Search for index.js and implied package exports #424
• support cycles between compartments Endo: Support cyclic dependencies across compartments #427 feat(endo): Support mutual dependency among compartments #433
• mask files absent in package.json "files" Endo: Files absent in package.json "files" should be unusable #426
• annotate JSON parse errors with source locations feat(endo): Improve JSON file parse error messages #430
• carry compartment names on error messages feat(ses): Carry compartment names in error messages #431
• Fix archive URI encoding mistake fix(endo): Do not URL encode compartment and path within archives #435
• Archive compartment maps should only capture the compartments and modules that are actually used Endo: Archived compartment map garbage collection #438
• distinguish module specifier events/ from events Endo: Distinguish "dependency" from "dependency/" module specifiers #425
• capturing lavamoat policies in compartment maps
• threading endowments to third-party compartments through compartment maps Endo: thread powers by policy #423
• expressing “replacement“ package overrides in “package.json”
• declaring worker entries in package.json and revealing worker constructors to compartments with named endowments
• supporting resources like XS new Resource(specifier)
• facilitate shims Vetted shims #422
• facilitate exits to built-in modules Endo: facilitate exits to built-ins and shims #439