Introduce swc_core, semver compliant rust package

[kwonoj]

Status	Last Updated	Target version
Accepted	2022/Jul/28	N/A

Summary

Provides compile time, runtime compatibility across various hosts and plugins compiled against different versions of swc_* packages.

Motivation

Currently, there are a handful of packages (or rusty term crates) published to allow to use SWC's core features. Major usecases for this packages are either creating a custom, native binary instead of using @swc/core to apply customizations (i.e https://nextjs.org/docs/advanced-features/compiler), or writing a plugin for the @swc/core which is a rust based codes requires to import certain feature of SWC.

Either you want to write a custom binary or a plugin, eventually this is typical examples of defining dependencies for the features of SWC. Each swc_* package intended to be a micro size module serves specific public interfaces to use. So there is no good way to avoid having multiple dependencies. Even though some dependencies are being used as transitive dependency - for example, pretty much most packages have reference to swc_common - if an application / plugin needs to import something directly from swc_common it has to be listed as direct dependency. The problem begins from here.

[dependencies]

swc = "0.205.0"
swc_atoms = "0.2.13"
swc_common = { version = "0.23.0", features = ["concurrent", "sourcemap"] }
swc_ecmascript = { version = "0.180.0", features = ["codegen", ...] }

# if you're writing a plugin
swc_plugin = { version = "0.82.0" }

...

Unlike node.js bindings (@swc/core), any rust packages do not honor semver and there is no clear guarantee certain version is compatible to others. Instead, we follow the moving forward model; the only thing we try to ensure is `if all the swc_* packages are latest, it should be built. While this wasn't cleanly understandable, it worked for most cases until we started to face cases we couldn't ensure those rules.

First, there are some chances certain transitive dependency's out of sync, so it conflicts with direct dependencies version.

swc_some = 0.64
 - swc_other
    - swc_inner = 0.63

// actual application
swc_some = 0.64
# now this conflicts to `swc_inner`
swc_inner = 0.64 

This one is more close to the bug actually, since SWC's publish should take care of bump up all the transitive dependencies altogether. But in reality this happens, and it happens more frequently as we move fast.

The second problem is bigger and not easily able to work around. If host & plugins have incompatible versions between 2, it'll cause a conflict either in build or runtime without clear message.

// @swc/core

swc_some = 0.64
swc_other = 0.64
 - swc_inner = 0.64

// my_plugin

swc_plugin = 0.63
 - swc_other = 0.63
 - swc_inner = 0.63

This is a simplified example of the version differences. This may work or not depends on the changes between 0.63 and 0.64. For this simplified example, the only dependency is swc_plugin which can be resolved easily. If a plugin gets complex and trying to involve more dependencies it exposes more collision points such as swc_plugin to others. Issues like swc-project/plugins#48 is caused by this.

More headache is to deal with multiple host runtimes. next/swc is a variant of SWC, which does not always have the same version / feature set as @swc/core.  If those 2 versions are incompatible, there's no way to make the plugin run on both hosts, or not even able to build it from the beginning (https://github.com/vercel/next.js/issues/38428). 

One of the main goals of the SWC's plugin system is having a single binary (wasm), runs on the supported platform as long as @swc/core reaches a major breaking change. With the current versioning system, it is not possible to achieve.

Lastly, there's a known discoverability issue as well. Since features are scattered around swc_*, it is unclear which package to be used for certain features. This isn't new and partially resolved by documentation though.

Detailed design

To avoid version conflicts across different packages, propose a new meta package swc_core. Package name is tentative, subject to change.

swc_core attempts to be served as isomorphic entry point to any features of SWC to the consumers. For writing custom native binary, or plugin, both needs to specify swc_core as a single dependency. It is a meta package that does not contain code on its own but only tries to reexport from transitive swc_* dependencies. Reexports are controlled by cargo's feature to avoid bringing in all the codes into compiled binary.

// pseudo swc_core/lib.rs

pub use swc_common::chain;
pub mod comments {
  pub use swc_common::comments::Comment
}

#[cfg(feature = plugin)]
pub use swc_common::plugin::PluginSerializedBytes

...

//pseudo swc_core/Cargo.toml

[features]

# all the features user may opt in, instead of manually importing each dependency

swc_ecmascript_minifier = []
swc_ecmascript_codegen = []
swc_common_sourcemap = []

Once it's completed, swc_core will try to honor semver. If there are any changes that different binaries rely on different versions of swc_core causes regression, it'll be considered breaking changes. Any other remaining packages will be treated as same. SWC will not prohibit the use of individual packages, but it can break anytime each time a new version's released user needs to resolve manually.

Drawbacks

This will bring interesting challenges to manage versions across dependencies to be reexported by meta package. Also, this creates a mega package that brings all the features into a single package, which potentially be a bottleneck for the build performance as well as raising tricky puzzles between non-compatible features as similar to plugin's wasm runtime.

Alternatives

We could try to ensure each individual package is compatible across versions. But this will be a more tricky challenge to verify various dependency combinations with possible transitive dependency conflicts.

Unresolved questions

There are some open questions that need to be resolved.

1. proc_macro and direct dependencies

It is unclear if we can prevent any additional imports to deps if proc_macro is involved. Similar to swc_plugin_macro probably we could make macro impl relies on the namespace of swc_core always. But this will be a large scale refactoring that requires careful approach to not break existing codebases.

2. Testing strategy

There is no good known deterministic testing we can say a new version is compatible to the old version.

3. Tangled inner dependency / feature tree

There are numbers of features exposed by each individual package. swc_core needs a way to expose it to public interface user can control. As there are a lot of tangled dependency trees internally, properly forwarding a specific feature from top to bottom may require sizable changes to existing feature / codebase. 

References (if exists)

#5149

Implementations

• feat(swc/core): introduce swc_core package #5364
• feat(swc/core): expand features, update ref to wasm / cli #5369
• Introduce `swc_core`, semver compliant rust package #5244

[kwonoj]

#5248 is an example internal version publish conflicts.

[kwonoj]

I think the biggest change for this one is proc_macro based packages, if they need to be exposed as public interface. As our goal is make users to avoid to reference separate packages, proc_macro packages need to reference swc_core:: 's reexported one instead of direct reference. This is already done for the plugins (i.e swc_plugin_macro always refer swc_plugin's exported, instead of referencing things directly) but other packages doesn't: may need some refactorings to make sure this work.

The other aspect is how do we have reliable tests to verify new PR / commit doesn't cause breaking changes. Other than having some simple compile-time codes, not sure if we can have extensive tests. Even removing features can be a breaking change.

[kwonoj]

I'll be changing this to accepted before working on this, but would love to get final approval stamp:

@kdy1 - thoughts, concerns, or objections?

[kdy1]

I love the idea, although I think we are currently following semver in a strict manner, (but with too many breaking changes)

[kwonoj]

So we did some discussions outside the RFC itself, summarizing here for the visibility.

Each swc_* package currently provides some sort of semver, but the major collision comes from inter-package version conflict. For the simplified example it looks like this:

• @swc/core built against swc_common, swc_ecma_xxx with version x+2,x
• swc_plugin built against swc_common, swc_ecma_xxx with version x+1, x
• user build plugin, imports swc_plugin@latest (which contains swc_common@x+1), then manually imports swc_common@x+2 for the imports not included in swc_plugin

Depends on the compatibility of x, x+1, x+2, it may work or not - but users don't have a good way to determine which version of swc_common they need to import, since swc_plugin doesn't transparently expose transitive dep's version / and also swc/core does not have a way to determine which version is compatible.

RFC tried to amend this by all 3 using single pkg (swc_core), and swc_core itself manages inter-pkg version to avoid this kind of conflict.

So far, we agreed to move this into accepted stage, and try to experiment actual behavior by making swc_core to work with plugins as first step.