Upgrade to TypeScript v5.0 and set module{,Resolution} option to Node16

[MajorLift]

Explanation

TypeScript v5.0

As part of the Wallet Framework Team's OKRs (Q2 2024 O3KR4, Q3 2024 O2KR4), we are upgrading TypeScript to v5.0+ for all packages in the core monorepo.

These upgrades will give us access to new features, aid us in writing more type-safe and modern code, and also allow us to reach parity with Extension and other MetaMask projects.

Node16

In order to maximize the benefits of this upgrade, we are also enabling the Node16 setting for the module and moduleResolution tsconfig options.

Motivation

Interop: CJS modules referencing ESM modules

The core monorepo is a collection of CJS packages, which use CJS module resolution rules internally, and are treated as CJS modules by Node.js. This is true despite that fact that these packages are authored in TypeScript using ESM syntax (import/export statements) and are set up to output dual builds for both CJS and ESM.

With Node16 or NodeNext enabled, CJS modules are unable to reference ESM modules via static/synchronous import statements, as TypeScript assumes them to be compiled down to CJS-only require statements.

There are three solutions for this issue, of which we are utilizing the first two:

1. Update the ESM-only dependency so that it outputs a CJS build and type declaration as well.
2. Replace the static import statements with dynamic import syntax (which, based on CJS emit rules, are not transformed to require statements).
3. Migrate our module to ESM (by setting "type": "module" in package.json and renaming all of our scripts to *.cjs)

See https://www.typescriptlang.org/docs/handbook/modules/reference.html#interoperability-rules

With dependencies that we control or use extensively, we pursue the first option, as we're doing with superstruct and @metamask/utils (and all of the many core dependencies that are downstream of either or both).

With dependencies that see more limited usage, we are opting for either the second option (e.g. multiformats) or, if available, using a CJS-compatible alternative (e.g. replacing lodash-es with lodash).

The third solution of migrating to ESM is the most fundamental, long-term measure, but we are refraining from it at this stage until we can make a concerted effort to migrate our codebase as a whole. This is because any individual ESM migration can cause a cascading effect through the dependency tree where other packages are now required to migrate as well.

Reasons for moving away from node/node10

The following outlines the motivation for switching to Node16, backed by relevant entries from the official TypeScript documentation.

• Starting with TypeScript v5.0, the previously used moduleResolution setting node is renamed to node10, and strongly discouraged from usage.
  • "It reflects the CommonJS module resolution algorithm as it existed in Node.js versions earlier than v12. It should no longer be used."
  • "Because node16 and nodenext are the only module options that reflect the complexities of Node.js’s dual module system, they are the only correct module options for all apps and libraries that are intended to run in Node.js v12 or later, whether they use ES modules or not."
• The node10 setting is unable to guarantee correct module resolution for ESM-only dependencies.
  • "Because Node.js v12 introduced different module resolution rules for ES modules, though, it’s a very bad model of modern versions of Node.js. It should not be used for new projects."
  • "node16 and nodenext describe the full range of behavior for Node.js’s dual-format module system, and emit files in either CommonJS or ESM format. This is different from every other module option, which are runtime-agnostic and force all output files into a single format, leaving it to the user to ensure the output is valid for their runtime."
• The node10 setting does not support the package.json "exports" field, which is used in our libraries to expose dual builds and type declarations.
  • https://www.typescriptlang.org/docs/handbook/modules/reference.html#packagejson-exports
• The Node16 and NodeNext settings maximize downstream compatibility.
  • "When compiling a library, you don’t know where the output code will run, but you’d like it to run in as many places as possible. Using "module": "nodenext" (along with the implied "moduleResolution": "nodenext") is the best bet for maximizing the compatibility of the output JavaScript’s module specifiers, since it will force you to comply with Node.js’s stricter rules for import module resolution."
  • ""moduleResolution": "nodenext" is only checking that the output works in Node.js, but in most cases, module code that works in Node.js will work in other runtimes and in bundlers"

Node16 vs. NodeNext

The two settings are currently identical, and Node16 is intended to work with all current node versions v16 or higher. If additional capabilities are added to NodeNext or Node18/Node20 that we want to apply to our codebases, we will be able to introduce them after checking for disruptive regressions or breaking changes.

Relationship with other tsconfig options

• --module nodenext or node16 implies and enforces the moduleResolution with the same name (and vice versa).
• --module node16 implies (up to) --target es2022.
  • --module nodenext implies (up to) --target esnext.

https://www.typescriptlang.org/docs/handbook/modules/reference.html#implied-and-enforced-options

Description

• Replace superstruct dependency with @metamask/superstruct ^3.0.0.
  • ^3.1.0
• Replace all superstruct import statements with @metamask/superstruct
• Bump @metamask/utils to ^8.5.0.
  • ^9.0.0
    • remove yarn resolution to @metamask/superstruct@npm:3.1.0
• Bump typescript to ~5.0.4
  • Set module and moduleResolution tsconfig options to Node16
  • Bump TypeScript module output options target and lib to ES2022 #4507

Further context on why the superstruct and utils changes are necessary:

• Use tsup for bundling utils#144
• Fix ESM/CJS compatibility superstruct#1
• Use ts-bridge as build tool, set moduleResolution to NodeNext superstruct#18
• Use ts-bridge for building utils#182
• Replace tsup with ts-bridge metamask-module-template#247
• https://www.typescriptlang.org/docs/handbook/modules/guides/choosing-compiler-options.html#considerations-for-bundling-libraries

Release order roadmap

Due to interdependencies between the packages involved in this PR, we will need to update and release them in a specific order:

• chore(deps): Bump @metamask/utils to ^9.0.0, @metamask/rpc-errors to ^6.3.1 #4516
• Release {base,permission}-controller
• (wait for releases: snaps-sdk, snaps-utils, snaps-controllers, keyring-api)
  • 8.0.1 keyring-api#356
  • Replace superstruct with ESM-compatible fork @metamask/superstruct snaps#2445
  • Release 56.0.0 snaps#2584
  • Release 57.0.0 snaps#2589
• Remove yarn resolutions for snaps-sdk, snaps-utils, keyring-api
• Merge this PR: Upgrade to TypeScript v5.0 and set module{,Resolution} option to Node16 #3645
  • Set yarn resolution for @metamask/providers via @metamask/snaps-sdk to 17.1.1
    • Leave messages in Changelog for affected packages ({accounts,chain,profile-sync}-controller) to hold off on new releases

    ### Uncategorized
    
    - Please hold off on new releases of this package until the yarn resolution for `@metamask/providers` is removed.
      - This is blocked by a `@metamask/snaps-sdk` release with `@metamask/providers` bumped to `>=17.1.1`.
        - See: [Fix regressions introduced by @metamask/providers@17.1.1](https://github.com/MetaMask/snaps/pull/2579)
    - Build error fixed by yarn resolution: [MetaMask/core/actions/runs/10011688901/job/27675682526?pr=3645](https://github.com/MetaMask/core/actions/runs/10011688901/job/27675682526?pr=3645)

• Release all core pkgs (especially deps of snaps-controllers and consumers of utils)
  • Exclude core pkgs that have snaps-controllers as dependency, and are affected by @metamask/providers yarn resolution
    • {accounts,chain,profile-sync}-controller
• Release snaps-controllers (Release 58.0.0 snaps#2595)
• Release {accounts,chain,profile-sync}-controller, eth-snap-keyring (chore(deps): replace superstruct imports with @metamask/superstruct eth-snap-keyring#311)
  • Bump @metamask/snaps-{controllers,sdk,utils} and remove yarn resolution for @metamask/providers #4547
  • Release 180.0.0 #4548
• (release remaining snaps packages while releasing @metamask/utils@9.0.0 version bumps for all dependencies and nested dependencies)

References

• Contributes to Upgrade to TypeScript v5.0 #3651
• Blocked by:
  • superstruct
    • Use ts-bridge as build tool, set moduleResolution to NodeNext superstruct#18
    • 3.0.0 superstruct#24
    • Add all types in the utils submodule as package-level exports superstruct#25
    • 3.1.0 superstruct#28
  • utils
    • Use @metamask/superstruct, set moduleResolution to NodeNext utils#185
    • 8.5.0 utils#191
    • Bump @metamask/superstruct from ^3.0.0 to ^3.1.0 utils#194
    • 9.0.0 utils#196
• Blocked by downstream consumers of superstruct, utils:
  • abi-utils:
    • Replace 'superstruct' imports with '@metamask/superstruct' abi-utils#73
    • 2.0.3 abi-utils#78
    • Bump @metamask/superstruct to ^3.1.0, @metamask/utils to ^9.0.0 abi-utils#80
    • 2.0.4 abi-utils#81
  • chain-api:
    • refactor: replace superstruct with @metamask/superstruct + bump @metamask/utils accounts-chain-api#5
    • https://github.com/MetaMask/accounts-chain-api/releases/tag/v0.1.0
  • eth-simple-keyring:
    • Bump @metamask/utils from ^8.1.0 to ^9.0.0 eth-simple-keyring#177
    • 6.0.2 eth-simple-keyring#178
  • providers:
    • Replace tsup with ts-bridge as build tool providers#336
    • 17.1.0 providers#337
      • Blocked by fix: ts-bridge import json error providers#340, Converting a json import that doesn't work as expected ts-bridge/ts-bridge#22
        • Causes CI failure: https://github.com/MetaMask/snaps/actions/runs/9783767567/job/27013136688?pr=2445
    • Bump @metamask/utils to ^9.0.0, bump @metamask/{json-rpc-engine,json-rpc-middleware-stream,rpc-errors} providers#345
    • 17.1.1 providers#347
  • rpc-errors:
    • chore(deps): Bump @metamask/utils from ^8.3.0 to ^9.0.0 rpc-errors#147
    • 6.3.1 rpc-errors#148
  • snaps-registry:
    • Replace 'superstruct' imports with '@metamask/superstruct' snaps-registry#613
    • 3.2.0 snaps-registry#670
    • Bump @metamask/superstruct to ^3.1.0, @metamask/utils to ^9.0.0 snaps-registry#693
    • 3.2.1 snaps-registry#694
• Blocked by snaps monorepo releases
  • snaps-sdk, snaps-utils
    • keyring-api
      • chore(deps): replace superstruct imports with @metamask/superstruct keyring-api#328
  • snaps-controllers
    • eth-snap-keyring
      • chore(deps): replace superstruct imports with @metamask/superstruct eth-snap-keyring#311

Changelog

@metamask/accounts-controller

### Changed

- Bump `@metamask/keyring-api` from `^8.0.0` to `^8.0.1` ([#3645](https://github.com/MetaMask/core/pull/3645))
- Bump `@metamask/snaps-sdk` from `^4.2.0` to `^6.1.0` ([#3645](https://github.com/MetaMask/core/pull/3645))
- Bump `@metamask/snaps-utils` from `^7.4.0` to `^7.8.0` ([#3645](https://github.com/MetaMask/core/pull/3645))
- Bump peer dependency `@metamask/snaps-controllers` from `^8.1.1` to `^9.3.0` ([#3645](https://github.com/MetaMask/core/pull/3645))

@metamask/assets-controllers (major)

### Changed

- **BREAKING:** `getIpfsCIDv1AndPath`, `getFormattedIpfsUrl` are now async functions ([#3645](https://github.com/MetaMask/core/pull/3645))
- Add `immer` `^9.0.6` as a new dependenc. ([#3645](https://github.com/MetaMask/core/pull/3645))
- Bump `@metamask/abi-utils` from `^2.0.2` to `^2.0.3` ([#3645](https://github.com/MetaMask/core/pull/3645))
- Bump `multiformats` from `^9.5.2` to `^13.1.0` ([#3645](https://github.com/MetaMask/core/pull/3645))

@metamask/chain-controller

### Changed

- Bump `@metamask/chain-api` from `^0.0.1` to `^0.1.0` ([#3645](https://github.com/MetaMask/core/pull/3645))
- Bump `@metamask/keyring-api` from `^8.0.0` to `^8.0.1` ([#3645](https://github.com/MetaMask/core/pull/3645))
- Bump `@metamask/snaps-controllers` from `^8.1.1` to `^9.3.0` ([#3645](https://github.com/MetaMask/core/pull/3645))
- Bump `@metamask/snaps-sdk` from `^4.2.0` to `^6.1.0` ([#3645](https://github.com/MetaMask/core/pull/3645))
- Bump `@metamask/snaps-utils` from `^7.4.0` to `^7.8.0` ([#3645](https://github.com/MetaMask/core/pull/3645))

@metamask/keyring-controller

### Changed

- Bump `@metamask/eth-simple-keyring` from `^6.0.1` to `^6.0.2` ([#3645](https://github.com/MetaMask/core/pull/3645))
- Bump `@metamask/keyring-api` from `^8.0.0` to `^8.0.1` ([#3645](https://github.com/MetaMask/core/pull/3645))
- Set yarn resolution for `@metamask/snaps-sdk/@metamask/providers` to `17.1.1` ([#3645](https://github.com/MetaMask/core/pull/3645))
  - Remove once `@metamask/snaps-sdk` bumps its `@metamask/providers` version to `>=17.1.1`.

@metamask/network-controller (minor)

### Added

- Newly exports the following types: `AutoManagedNetworkClient`, `InfuraNetworkClientConfiguration`, `CustomNetworkClientConfiguration` ([#3645](https://github.com/MetaMask/core/pull/3645))

@metamask/profile-sync-controller

### Changed

- Bump dependency and peer dependency `@metamask/snaps-controllers` from `^8.1.1` to `^9.3.0` ([#3645](https://github.com/MetaMask/core/pull/3645))
- Bump `@metamask/snaps-sdk` from `^4.2.0` to `^6.1.0` ([#3645](https://github.com/MetaMask/core/pull/3645))
- Bump `@metamask/snaps-utils` from `^7.4.0` to `^7.8.0` ([#3645](https://github.com/MetaMask/core/pull/3645))

@metamask/transaction-controller

### Changed

- Bump `@metamask/keyring-api` from `^8.0.0` to `^8.0.1` ([#3645](https://github.com/MetaMask/core/pull/3645))

@metamask/user-operation-controller

### Fixed

- Replace `superstruct` with ESM-compatible `@metamask/superstruct` `^3.1.0` ([#3645](https://github.com/MetaMask/core/pull/3645))
  - This fixes the issue of this package being unusable by any TypeScript project that uses `Node16` or `NodeNext` as its `moduleResolution` option.

Checklist

• I've updated the test suite for new or updated code as appropriate
• I've updated documentation (JSDoc, Markdown, etc.) for new or updated code as appropriate
• I've highlighted breaking changes using the "BREAKING" category above as appropriate

[mcmire]

I worked on this a bit and by bumping some packages and also using a local copy of utils which makes use of @metamask/superstruct instead of superstruct, I managed to get yarn build:types working. I created a separate branch and made a commit there with the necessary changes, and you can see the package bumps here: df14ce8. You'll want to use your own local copy of utils with the use-metamask-superstruct branch checked out (make sure to run yarn build first).

I haven't tried running yarn test yet, so I don't know if that works. I will try that next.

[socket-security]

👍 Dependency issues cleared. Learn more about Socket for GitHub ↗︎

This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored.

Ignoring: npm/micromatch@4.0.7, npm/node-gyp@10.2.0, npm/validate-npm-package-name@5.0.1

View full report↗︎

Next steps

Take a deeper look at the dependency

Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support [AT] socket [DOT] dev.

Remove the package

If you happen to install a dependency that Socket reports as Known Malware you should immediately remove it and select a different dependency. For other alert types, you may may wish to investigate alternative packages or consider if there are other ways to mitigate the specific risk posed by the dependency.

Mark a package as acceptable risk

To ignore an alert, reply with a comment starting with @SocketSecurity ignore followed by a space separated list of ecosystem/package-name@version specifiers. e.g. @SocketSecurity ignore npm/foo@1.0.0 or ignore all packages with @SocketSecurity ignore-all

[MajorLift]

Notes for reviewers

[MajorLift]

multiformats is an ESM-only package, and we are replacing its import statement with dynamic import syntax.

This causes the following changes:

• getIpfsCIDv1AndPath, getFormattedIpfsUrl are now async functions (this is a breaking change for assets-controllers).
• The --experimental-vm-modules flag needs to be prepended to our build scripts that run jest, with accompanying updates to our yarn constraints.

[Mrtenz]

How did this work previously if it's ESM-only?

[MajorLift]

So far as I can tell, our previous moduleResolution option node/node10 will happily pass ESM-only packages into require calls without raising any warnings.

So it's likely the warnings we're now seeing with node16 should have been there all along. The package migrated to ESM 4 years ago (multiformats/js-multiformats@f925195#diff-7ae45ad102eab3b6d7e7896acd08c427a9b25b346470d7bc6507b6481575d519).

At runtime, either the module was interoperable by coincidence (not quite sure if this is possible or likely -- maybe if there's no ESM-specific syntax?), or it was breaking and we didn't notice.

[MajorLift]

The --experimental-vm-modules flag is necessary if there are dynamic imports in the compiled code used by our tests.

[MajorLift]

These exports accommodate the new import statements in Token{Detection,Rate}Controller.test.ts.

Previously, these types were accessed through subpath imports, which are now not supported unless specified in the library's package.json exports field.

[MajorLift]

Only listed as a devDep in the root package.json, where it's used in a build script but not in any modules.

[MajorLift]

With the Node{16,Next} option, subpath imports are no longer supported unless explicitly defined in the source package manifest's "exports" field.

Here, an import from @metamask/accounts-controller/src/tests/mocks needed to be replaced with a relative path import.

[mcmire]

Ah I'm not sure how this slipped through the cracks! Makes sense.

[MajorLift]

• Fixes build error due to package version mismatch: MetaMask/core/actions/runs/10011688901/job/27675682526?pr=3645
• Removal is blocked by a @metamask/snaps-sdk release with @metamask/providers bumped to >=17.1.1.
  • See: Fix regressions introduced by @metamask/providers@17.1.1

[MajorLift]

The packages affected by the yarn resolution are {accounts,chain,profile-sync}-controller. It's safe to temporarily merge the yarn resolution into main, as long as these packages aren't released (and even if they are @metamask/providers is not a direct dependency of any of these packages).

In any case, these three packages will be excluded from the core releases that follow this PR, since they are blocked by a snaps-controllers release as well.

[mcmire]

I obviously underestimated how long this would take. Fantastic work on keeping track and staying on top of all of the downstream upgrades in order to unlock this PR 🎉

[MajorLift]

For the @MetaMask/confirmations team: this is the only change in this PR to a confirmations-owned package (aside from the typescript bumps and test build script updates on every package).