Skip to content

Latest commit

 

History

History
218 lines (172 loc) · 9.34 KB

008-npm-dependencies.md

File metadata and controls

218 lines (172 loc) · 9.34 KB
Raw
  • Start Date: 2018-02-14
  • RFC PR: (leave this empty)
  • Tracking Issue: (leave this empty)

Summary

Enable Rust crates to transparently depend on packages in the npm ecosystem. These dependencies will, like normal Rust dependencies through Cargo, work seamlessly when consumed by other crates.

Motivation

The primary goal of wasm-bindgen and wasm-pack is to enable seamless integration of Rust with JS. A massive portion of the JS ecosystem, npm, however currently has little support in wasm-bindgen and wasm-pack, making it difficult to access this rich resource that JS offers!

The goal of this RFC is to enable these dependencies to exist. Rust crates should be able to require functionality from NPM, just like how NPM can require Rust crates compiled to wasm. Any workflow which currently uses NPM packages (such as packaging WebAssembly with a bundler) should continue to work but also allow pulling in "custom" NPM packages as well as requested by Rust dependencies.

Stakeholders

This RFC primarily affects uses of wasm-pack and wasm-bindgen who are also currently using bundlers like Webpack. This also affects, however, developers of core foundational crates in the Rust ecosystem who want to be concious of the ability to pull in NPM dependencies and such.

Detailed Explanation

Adding an NPM dependency to a Rust project will look very similar to adding an NPM dependency to a normal JS project. First the dependency, and its version requirement, need to be declare. This RFC proposes doing this in a package.json file adjacent to the crate's Cargo.toml file:

 {
  "dependencies": {
    "foo": "^1.0.1"
  }
}

The package.json file will initially be a subset of NPM's package.json, only supporting one dependencies top-level key which internally has key/value pairs with strings. Beyond this validation though no validation will be performed of either key or value pairs within dependencies. In the future it's intended that more keys of package.json in NPM will be supported, but this RFC is intended to be an MVP for now to enable dependencies on NPM at all.

After this package.json file is created, the package next needs to be imported in the Rust crate. Like with other Rust dependencies on JS, this will be done with the #[wasm_bindgen] attribute:

#[wasm_bindgen(module = "foo")]
extern "C" {
    fn function_in_foo_package();
}

Note: in JS the above import would be similar to:

import { function_in_foo_package } from "foo";

The exiting module key in the #[wasm_bindgen] attribute can be used to indicate which ES module the import is coming from. This affects the module key in the final output wasm binary, and corresponds to the name of the package in package.json. This is intended to match how bundler conventions already interpret NPM packages as ES modules.

After these two tools are in place, all that's needed is a wasm-pack build and you should be good to go! The final package.json will have the foo dependency listed in our package.json above and be ready for consumption via a bundler.

Technical Implementation

Under the hood there's a few moving parts which enables all of this to happen. Let's first take a look at the pieces in wasm-bindgen.

The primary goal of this RFC is to enable tranparent and transitive dependencies on NPM. The #[wasm_bindgen] macro is the only aspect of a crate's build which has access to all transitive dependencies, so this is what we'll be using to slurp up package.json. When #[wasm_bindgen] with a module key is specified it will look for package.json inside the cwd of the procedural macro (note that the cwd is set by Cargo to be the directory with the crate's Cargo.toml that is being compiled, or the crate in which #[wasm_bindgen] is written). This package.json, if found, will have an absolute path to it encoded into the custom section that wasm-bindgen already emits.

Later, when the wasm-bindgen CLI tool executes, it will parse an interpret all items in the wasm-bindgen custom section. All package.json files listed will be loaded, parsed, and validated (aka only dependencies allowed for now). If any package.json is loaded then a package.json file will be emitted next to the output JS file inside of --out-dir.

After wasm-bindgen executes, then wasm-pack will read the package.json output, if any, and augment it with metadata and other items which are already emitted.

If more than one crate in a dependency graph depends on an NPM package then in this MVP proposal an error will be generated. In the future we can implement some degree of merging version requirements, but for now to remain simple wasm-bindgen will emit an error.

Interaction with --no-modules

Depending on NPM packages fundamentally requires, well, NPM, in one way or another. The wasm-bindgen and wasm-pack CLI tools have modes of output (notably wasm-bindgen's --no-modules and wasm-pack's --target no-modules flags) which are intended to not require NPM and other JS tooling. In these situations if a package.json in any Rust crate is detected an error will be emitted indicating so.

Note that this means that core crates which are intended to work with --no-modules will not be able add NPM dependencies. Instead they'll have to either import Rust dependencies from crates.io or use a feature like local JS snippets to import custom JS code.

Drawbacks

One of the primary drawbacks of this RFC is that it's fundamentally incompatible with a major use case of wasm-bindgen and wasm-pack, the --no-modules and --target no-modules flags. As a short-term band-aid this RFC proposes making it a hard error which would hinder the adoption of this feature in crates that want to be usable in this mode.

In the long-term, however, it may be possible to get this working. For example many NPM packages are available on unpkg.com or in other locations. It may be possible, if all packages in these locations adhere to well-known conventions, to generate code that's compatible with these locations of hosting NPM packages. In these situations it may then be possible to "just drop a script tag" in a few locations to get --no-modules working with NPM packages. It's unclear how viable this is, though.

Rationale and Alternatives

When developing this RFC, some guiding values for its design have been articulated:

  • Development on Rust-generated WebAssembly projects should allow developers to use the development environment they are most comfortable with. Developers writing Rust should get to use Rust, and developers using JavaScript should get to use a JS based runtime environment (Node.js, Chakra, etc).

  • JavaScript tooling and workflows should be usable with Rust-generated WebAssembly projects. For example, bundlers like WebPack and Parcel, or dependency management tools such as npm audit and GreenKeeper.

  • When possible, decisions should be made that allow the solution to be available to developers of not just Rust, but also C, and C++.

  • Decisions should be focused on creating workflows that allow developers an easy learning curve and productive development experience.

These principles lead to the above proposal of using package.json to declare NPM dependencies which is then grouped together by wasm-bindgen to be published by wasm-pack. By using package.json we get inherent compatibility with existing workflows like GreenKeeper and npm install. Additionally package.json is very well documented and supported throughout the JS ecosystem making it very familiar.

Some other alternatives to this RFC which have been ruled out are:

  • Using Cargo.toml instead of package.json to declare NPM dependencies. For example we could use:

    [package.metadata.npm.dependencies]
foo = "0.1"

    This has the drawback though of being incompatible with all existing workflows around package.json. Additionally it also highlights a discrepancy between NPM and Cargo and how "0.1" as a version requirement is interpreted (e.g. ^0.1 or ~0.1).

  • Adding a separate manifest file instead of using package.json is also possibility and might be easier for wasm-bindgen to read and later parse/include. This has a possible benefit of being scoped to exactly our use case and not being misleading by disallowing otherwise-valid fields of package.json. The downside of this approach is the same as Cargo.toml, however, in that it's an unfamiliar format to most and is incompatible with existing tooling without bringing too much benefit.

  • Annotating version dependencies inline could be used rather than package.json as well, such as:

    #[wasm_bindgen(module = "foo", version = "0.1")]
extern "C" {
    // ...
}

    As with all other alternatives this is incompatible with existing tooling, but it's also not aligned with Rust's own mechanism for declaring dependencies which separates the location for version information and the code iteslf.

Unresolved Questions

  • Is the MVP restriction of only using dependencies too limiting? Should more fields be supported in package.json?