feat: async modules / top level await #5450

ForsakenHarmony · 2023-07-03T21:10:32Z

Description

For now this is just for top level await, in the future we want to be able to support external esm modules

Testing Instructions

vercel · 2023-07-03T21:10:41Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
examples-designsystem-docs	🔄 Building (Inspect)	Visit Preview	💬 Add feedback	Jul 20, 2023 2:39pm
examples-kitchensink-blog	🔄 Building (Inspect)	Visit Preview	💬 Add feedback	Jul 20, 2023 2:39pm
examples-native-web	🔄 Building (Inspect)	Visit Preview	💬 Add feedback	Jul 20, 2023 2:39pm
examples-vite-web	🔄 Building (Inspect)	Visit Preview	💬 Add feedback	Jul 20, 2023 2:39pm

7 Ignored Deployments

Name	Status	Preview	Updated (UTC)
examples-basic-web	⬜️ Ignored (Inspect)	Visit Preview	Jul 20, 2023 2:39pm
examples-cra-web	⬜️ Ignored (Inspect)	Visit Preview	Jul 20, 2023 2:39pm
examples-gatsby-web	⬜️ Ignored (Inspect)	Visit Preview	Jul 20, 2023 2:39pm
examples-nonmonorepo	⬜️ Ignored (Inspect)	Visit Preview	Jul 20, 2023 2:39pm
examples-svelte-web	⬜️ Ignored (Inspect)	Visit Preview	Jul 20, 2023 2:39pm
examples-tailwind-web	⬜️ Ignored (Inspect)	Visit Preview	Jul 20, 2023 2:39pm
turbo-site	⬜️ Ignored (Inspect)	Visit Preview	Jul 20, 2023 2:39pm

crates/turbopack-ecmascript-runtime/js/src/shared/runtime-utils.ts

github-actions · 2023-07-03T21:15:06Z

✅ This change can build next-swc

github-actions · 2023-07-03T21:34:12Z

⚠️ CI failed ⚠️

The following steps have failed in CI:

Turbopack Rust tests (mac/win, non-blocking)

See workflow summary for details

sokra

Looks good in general.

I'm not sure if this hangs when there are cycles in the module graph... need to verify, maybe with a test

It would also be great to have some tests that actually evaluate the code. Wills PR would help here

crates/turbopack-ecmascript-runtime/js/src/shared/runtime-utils.ts

sokra · 2023-07-05T14:54:46Z

crates/turbopack-ecmascript/src/analyzer/top_level_await.rs

    let mut visitor = TopLevelAwaitVisitor::default();

    m.visit_with(&mut visitor);

-    visitor.has_top_level_await
+    visitor.top_level_await_span
 }

 #[derive(Default)]
 struct TopLevelAwaitVisitor {


It might be possible to merge this visitor into detect_dynamic_export to avoid walking the ast multiple times

sokra

Ok there are a bunch of new interactions that didn't exist with webpack.

We expose 2 types of exports module.exports (used for require) and module.namespaceObject (use for import)
For an non-ESM (where module.namespaceObject doesn't exist), we use interopEsm to create a interop namespace object.
For ESM module.exports and module.namespaceObject must be equal
We need to proxify the exports for dynamicExport. (Both module.exports and module.namespaceObject)
We need to promisify the exports for async modules.
and 5. are quite new.

First thing: It seems like dynamicExport is a bit wrong (even on main) as it doesn't proxify module.exports. We need to add that.

Second thing: When combining 4. and 5. we need to proxify the resolved value of the promise. But we also don't want to change the Promise instance since it could already be imported (and awaited) by some other modules.

So we want to be able to change the resolved value.

So the dynamicImport method probably wants to detect async modules and modify the module.namespaceObject[turbopackExports] value. And the async modules implementation wants to avoid using it's local scoped exports object and instead resolve to the current value of promise[turbopackExports], which might be proxified.

crates/turbopack-ecmascript-runtime/js/src/shared/runtime-utils.ts

github-actions · 2023-07-20T14:57:36Z

Linux Benchmark for 945ba0b

Click to view benchmark

Test	Base	PR	%
bench_hmr_to_commit/Turbopack CSR/1000 modules	7588.84µs ± 24.87µs	7488.74µs ± 42.07µs	-1.32%
bench_hmr_to_eval/Turbopack CSR/1000 modules	6842.72µs ± 15.45µs	6927.77µs ± 75.00µs	+1.24%
bench_startup/Turbopack CSR/1000 modules	871.09ms ± 5.06ms	863.18ms ± 4.54ms	-0.91%

sokra

I'll merge it, but there are some outstanding follow-up tasks:

Add documentation to all public items. Especially in the async_module.rs file. Mention what Scc stands for or maybe even write out the full name.
Add a test case of circular dependency graphs.

* vercel/turborepo#5567  * vercel/turborepo#5576  * vercel/turborepo#5503  * vercel/turborepo#5487  * vercel/turborepo#5488  * vercel/turborepo#5450  --------- Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>

### Description For now this is just for top level await, in the future we want to be able to support external esm modules ### Testing Instructions Closes WEB-434 Closes WEB-987

ForsakenHarmony requested a review from a team as a code owner July 3, 2023 21:10

turbo-orchestrator bot added the team: turbopack label Jul 3, 2023

ForsakenHarmony commented Jul 3, 2023

View reviewed changes

crates/turbopack-ecmascript-runtime/js/src/shared/runtime-utils.ts Show resolved Hide resolved

ForsakenHarmony force-pushed the hrmny/web-434-async-modules branch 2 times, most recently from 4e42c42 to 3ba7bf7 Compare July 3, 2023 21:24

ForsakenHarmony force-pushed the hrmny/web-434-async-modules branch 2 times, most recently from 172177c to 586e769 Compare July 4, 2023 17:31

sokra requested changes Jul 5, 2023

View reviewed changes

ForsakenHarmony requested a review from sokra July 5, 2023 18:24

sokra requested changes Jul 5, 2023

View reviewed changes

crates/turbopack-ecmascript-runtime/js/src/shared/runtime-utils.ts Outdated Show resolved Hide resolved

crates/turbopack-ecmascript-runtime/js/src/shared/runtime-utils.ts Outdated Show resolved Hide resolved

ForsakenHarmony force-pushed the hrmny/web-434-async-modules branch from 2099396 to d2cfc7c Compare July 6, 2023 15:13

vercel bot deployed to Preview – turbo-site July 6, 2023 15:14 View deployment

ForsakenHarmony force-pushed the hrmny/web-434-async-modules branch 5 times, most recently from dd681f7 to 41f9e62 Compare July 10, 2023 15:53

vercel deleted a comment from github-actions bot Jul 10, 2023

vercel bot deployed to Preview – turbo-site July 12, 2023 18:44 View deployment

ForsakenHarmony force-pushed the hrmny/web-434-async-modules branch from e49643d to 6052d5d Compare July 12, 2023 18:58

ForsakenHarmony added 2 commits July 20, 2023 15:37

simplify dynamicExport

a4a0e21

unsimplify dynamicExport

5556ebd

ForsakenHarmony force-pushed the hrmny/web-434-async-modules branch from b3439b4 to 5556ebd Compare July 20, 2023 13:38

vercel bot deployed to Preview – turbo-site July 20, 2023 13:39 View deployment

vercel deleted a comment from github-actions bot Jul 20, 2023

revert snapshot changes

3b0be02

vercel deleted a comment from github-actions bot Jul 20, 2023

glob fixes, though it seems it's fully recursive

732dd01

ForsakenHarmony requested a review from sokra July 20, 2023 15:31

sokra approved these changes Jul 21, 2023

View reviewed changes

sokra merged commit fc4cfaa into main Jul 21, 2023
35 of 37 checks passed

sokra deleted the hrmny/web-434-async-modules branch July 21, 2023 06:53

sokra mentioned this pull request Jul 21, 2023

update Turbopack vercel/next.js#52986

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: async modules / top level await #5450

feat: async modules / top level await #5450

ForsakenHarmony commented Jul 3, 2023

vercel bot commented Jul 3, 2023 •

edited

Loading

github-actions bot commented Jul 3, 2023 •

edited

Loading

github-actions bot commented Jul 3, 2023 •

edited

Loading

sokra left a comment

sokra Jul 5, 2023

sokra left a comment

github-actions bot commented Jul 20, 2023

sokra left a comment

feat: async modules / top level await #5450

feat: async modules / top level await #5450

Conversation

ForsakenHarmony commented Jul 3, 2023

Description

Testing Instructions

vercel bot commented Jul 3, 2023 • edited Loading

github-actions bot commented Jul 3, 2023 • edited Loading

github-actions bot commented Jul 3, 2023 • edited Loading

⚠️ CI failed ⚠️

sokra left a comment

Choose a reason for hiding this comment

sokra Jul 5, 2023

Choose a reason for hiding this comment

sokra left a comment

Choose a reason for hiding this comment

github-actions bot commented Jul 20, 2023

Linux Benchmark for 945ba0b

sokra left a comment

Choose a reason for hiding this comment

vercel bot commented Jul 3, 2023 •

edited

Loading

github-actions bot commented Jul 3, 2023 •

edited

Loading

github-actions bot commented Jul 3, 2023 •

edited

Loading