Do not transpile JS packages twice #44824

scinos · 2020-08-11T10:51:54Z

Summary

Do not transpile some packages at installation time because webpack/jest will transpile them anyway. This allows faster installations (~38% faster) and smaller bundles.

Based on #43679

Background

Packages in ./packages/ implement a prepare script that is run every time we do yarn install. For JavaScript packages, this script is using babel to transpile the source to CJS and ESM. Those packages expose the ESM result in pkg.module (used by Webpack) and the CJS result in pkg.main (used by Node).

Later, when compiling ./client, webpack loads the ESM transpiled files, transpiles them again and add them to the bundles.

Changes

Do not implement a prepare script. Instead, use the prepack script to transpile to CJS. This will run the transpilation when yarn publish or yarn pack, but not on installation (see https://docs.npmjs.com/misc/scripts)
Change module to point to the source code (not the transpiled ESM). This allows distributing the untranspiled source code so consumers of this package (including Calypso) have control over the transpilation options. Specifically, this allows Calypso to transpile them for "evergreen" browsers. Before this change, packages in ./packages were always transpiled for "fallback" browsers.
Drop the prepare script. This will allow faster yarn build-packages (which is run on every yarn install). In my machine, in master yarn build-packages takes 39.9s to "prepare" 23 packages. After this change, it takes 24.6s (38% faster) to prepare 10 packages.

Testing instructions

Verify calypso.live loads (both evergreen and fallback) and there are no noticeable errors. If there is something broken entire sections will fail to load, so it should be easy to spot.
Verify that ICFY reports no size changes (so no new packages are added to the bundle).
Verify that tests passes (so Jest also understand the changes).

Update (22/Oct)

Instead of changing the meaning of module, this PR introduces a new calypso:src field that points to the untranspiled source code. pkg.module and pkg.main are unchanged.

matticbot · 2020-08-11T10:51:56Z

Test live: https://calypso.live/?branch=try/dont-transpile-packages

matticbot · 2020-08-11T10:55:19Z

Here is how your PR affects size of JS and CSS bundles shipped to the user's browser:

App Entrypoints (~4082 bytes removed 📉 [gzipped])

name                   parsed_size           gzip_size
entry-main                -11132 B  (-0.8%)    -1268 B  (-0.4%)
entry-login               -10924 B  (-1.2%)    -1189 B  (-0.5%)
entry-domains-landing     -10176 B  (-1.8%)    -1127 B  (-0.7%)
entry-gutenboarding        -6809 B  (-0.4%)     -498 B  (-0.1%)

Common code that is always downloaded and parsed every time the app is loaded, no matter which route is used.

Sections (~2951 bytes added 📈 [gzipped])

name                    parsed_size           gzip_size
jetpack-connect             +2893 B  (+0.3%)     +600 B  (+0.2%)
signup                      +2807 B  (+0.6%)     +544 B  (+0.4%)
gutenberg-editor            +2807 B  (+0.5%)     +549 B  (+0.3%)
accept-invite               +2708 B  (+0.5%)     +542 B  (+0.4%)
zoninator                   +2451 B  (+0.8%)     +548 B  (+0.6%)
checkout                    +2195 B  (+0.1%)     +473 B  (+0.1%)
purchase-product            +2194 B  (+0.9%)     +414 B  (+0.6%)
domains                     +1879 B  (+0.2%)     +533 B  (+0.2%)
migrate                     +1340 B  (+0.9%)     +309 B  (+0.7%)
jetpack-cloud-pricing       +1305 B  (+0.4%)     +305 B  (+0.3%)
posts-custom                -1111 B  (-0.3%)     -296 B  (-0.3%)
posts                       -1111 B  (-0.3%)     -295 B  (-0.3%)
plans                       +1110 B  (+0.1%)     +229 B  (+0.1%)
site-purchases              +1070 B  (+0.1%)     +290 B  (+0.1%)
purchases                    +985 B  (+0.1%)     +297 B  (+0.1%)
woocommerce                  -612 B  (-0.0%)      -84 B  (-0.0%)
reader                       -557 B  (-0.1%)      -98 B  (-0.1%)
activity                     -542 B  (-0.1%)     -112 B  (-0.1%)
media                        -511 B  (-0.1%)     -104 B  (-0.1%)
settings                     -497 B  (-0.1%)     -112 B  (-0.1%)
marketing                    -491 B  (-0.1%)     -131 B  (-0.1%)
themes                       -482 B  (-0.1%)     -108 B  (-0.1%)
stats                        -462 B  (-0.1%)     -109 B  (-0.1%)
backup                       -462 B  (-0.1%)      -96 B  (-0.1%)
settings-writing             -457 B  (-0.1%)     -105 B  (-0.1%)
settings-performance         -452 B  (-0.1%)     -114 B  (-0.1%)
plugins                      -452 B  (-0.1%)      -99 B  (-0.1%)
theme                        -442 B  (-0.1%)     -104 B  (-0.1%)
settings-security            -437 B  (-0.1%)     -109 B  (-0.1%)
email                        -437 B  (-0.1%)      -75 B  (-0.1%)
earn                         -412 B  (-0.1%)      -97 B  (-0.1%)
hosting                      -382 B  (-0.2%)      -94 B  (-0.1%)
security                     -269 B  (-0.1%)      -13 B  (-0.0%)
help                         -225 B  (-0.0%)      -20 B  (-0.0%)
account                      -190 B  (-0.1%)       -5 B  (-0.0%)
google-my-business           -179 B  (-0.1%)      -27 B  (-0.0%)
home                         -175 B  (-0.0%)      -24 B  (-0.0%)
comments                     -175 B  (-0.0%)      -21 B  (-0.0%)
account-close                -170 B  (-0.0%)       -6 B  (-0.0%)
scan                         -165 B  (-0.0%)      -19 B  (-0.0%)
settings-jetpack             -160 B  (-0.1%)      -20 B  (-0.0%)
people                       -155 B  (-0.0%)      -17 B  (-0.0%)
notification-settings        -155 B  (-0.0%)       -4 B  (-0.0%)
site-blocks                  -145 B  (-0.0%)       +2 B  (+0.0%)
settings-discussion          -140 B  (-0.1%)      -20 B  (-0.0%)
privacy                      -140 B  (-0.1%)       +3 B  (+0.0%)
happychat                    -140 B  (-0.0%)       -2 B  (-0.0%)
concierge                    -140 B  (-0.0%)      -11 B  (-0.0%)
me                           -135 B  (-0.1%)       -1 B  (-0.0%)
import                       -120 B  (-0.1%)      -16 B  (-0.0%)
wp-super-cache               -115 B  (-0.0%)       -9 B  (-0.0%)
pages                        -115 B  (-0.0%)      -14 B  (-0.0%)
export                       -115 B  (-0.0%)      -15 B  (-0.0%)
jetpack-cloud-settings       -105 B  (-0.1%)      -14 B  (-0.0%)
hello-dolly                   -80 B  (-0.1%)       -5 B  (-0.0%)
customize                     -80 B  (-0.0%)      -20 B  (-0.0%)
preview                       -75 B  (-0.0%)      -13 B  (-0.0%)
sensei                        -70 B  (-0.1%)       -7 B  (-0.0%)
jetpack-cloud                 -60 B  (-0.1%)       -9 B  (-0.0%)
sites                         -55 B  (-0.1%)       -7 B  (-0.0%)
devdocs                       -40 B  (-0.0%)       -4 B  (-0.0%)
auth                          -20 B  (-0.1%)       -2 B  (-0.0%)

Sections contain code specific for a given set of routes. Is downloaded and parsed only when a particular route is navigated to.

Async-loaded Components (~482 bytes removed 📉 [gzipped])

name                                                                 parsed_size            gzip_size
async-load-calypso-my-sites-current-site-notice                          -9616 B  (-15.4%)    -1746 B  (-12.0%)
async-load-design                                                        +2977 B   (+0.2%)     +688 B   (+0.2%)
async-load-design-wordpress-components-gallery                           +2872 B   (+0.3%)     +557 B   (+0.3%)
async-load-calypso-components-web-preview-component                      -2283 B   (-0.4%)     -509 B   (-0.3%)
async-load-design-playground                                             +1999 B   (+0.1%)     +481 B   (+0.1%)
async-load-design-blocks                                                 +1749 B   (+0.1%)     +650 B   (+0.1%)
async-load-signup-steps-site-topic                                        +712 B   (+2.5%)     +186 B   (+2.3%)
async-load-signup-steps-about                                             +697 B   (+1.2%)     +187 B   (+1.1%)
async-load-calypso-blocks-editor-checkout-modal                           -502 B   (-0.0%)     -112 B   (-0.0%)
async-load-calypso-post-editor-editor-media-modal                         -461 B   (-0.1%)     -100 B   (-0.1%)
async-load-signup-steps-plans                                             -402 B   (-0.2%)      -95 B   (-0.2%)
async-load-signup-steps-domains                                           -382 B   (-0.1%)      -93 B   (-0.1%)
async-load-signup-steps-plans-atomic-store                                -332 B   (-0.2%)      -90 B   (-0.2%)
async-load-signup-steps-secure-your-brand                                 -267 B   (-1.1%)      -84 B   (-1.5%)
async-load-calypso-blocks-jitm-templates-sidebar-banner                   -262 B   (-0.9%)      -84 B   (-1.2%)
async-load-calypso-blocks-jitm-templates-notice                           -262 B   (-0.9%)      -84 B   (-1.2%)
async-load-calypso-blocks-jitm-templates-default                          -262 B   (-1.0%)      -84 B   (-1.2%)
async-load-calypso-post-editor-media-modal                                -169 B   (-0.1%)      -15 B   (-0.0%)
async-load-calypso-blocks-inline-help-popover                             -145 B   (-0.0%)      -27 B   (-0.0%)
async-load-signup-steps-user                                              -119 B   (-0.1%)       -4 B   (-0.0%)
async-load-signup-steps-clone-point                                        -75 B   (-0.0%)       -4 B   (-0.0%)
async-load-calypso-extensions-woocommerce-app-store-stats-referrers        -60 B   (-0.1%)       +0 B
async-load-calypso-extensions-woocommerce-app-store-stats                  -55 B   (-0.1%)       +0 B
async-load-signup-steps-clone-credentials                                  -50 B   (-0.1%)       -3 B   (-0.0%)
async-load-calypso-my-sites-sidebar                                        -50 B   (-0.0%)      -10 B   (-0.0%)
async-load-calypso-blocks-inline-help                                      -45 B   (-0.0%)       -3 B   (-0.0%)
async-load-signup-steps-theme-selection                                    -40 B   (-0.1%)       +3 B   (+0.0%)
async-load-signup-steps-import-url-onboarding                              -40 B   (-0.1%)       -4 B   (-0.0%)
async-load-quick-language-switcher                                         -40 B   (-0.1%)       -6 B   (-0.0%)
async-load-calypso-reader-following-manage                                 -40 B   (-0.0%)       -5 B   (-0.0%)
async-load-signup-steps-site-picker                                        -35 B   (-0.1%)       -5 B   (-0.0%)
async-load-components-sites-popover                                        -35 B   (-0.1%)       -4 B   (-0.0%)
async-load-calypso-reader-sidebar                                          -35 B   (-0.1%)       +1 B   (+0.0%)
async-load-calypso-my-sites-sidebar-unified                                -35 B   (-0.1%)       -4 B   (-0.0%)
async-load-calypso-extensions-woocommerce-app-store-stats-listview         -35 B   (-0.2%)       -1 B   (-0.0%)
async-load-signup-steps-rewind-form-creds                                  -30 B   (-0.1%)       -3 B   (-0.0%)
async-load-signup-steps-import-url                                         -30 B   (-0.1%)       -1 B   (-0.0%)
async-load-calypso-reader-site-stream                                      -30 B   (-0.1%)       -8 B   (-0.1%)
async-load-calypso-reader-feed-stream                                      -30 B   (-0.1%)       -5 B   (-0.1%)
async-load-calypso-blocks-support-article-dialog-dialog                    -30 B   (-0.0%)       -2 B   (-0.0%)
async-load-signup-steps-p2-site                                            -25 B   (-0.1%)       -2 B   (-0.0%)
async-load-signup-steps-import-preview                                     -25 B   (-0.1%)       -3 B   (-0.0%)
async-load-calypso-reader-search-stream                                    -25 B   (-0.0%)       -2 B   (-0.0%)
async-load-calypso-my-sites-guided-transfer                                -25 B   (-0.1%)       +0 B
async-load-calypso-blocks-reader-full-post                                 -25 B   (-0.1%)       -2 B   (-0.0%)
async-load-signup-steps-site                                               -20 B   (-0.1%)       -3 B   (-0.0%)
async-load-signup-steps-clone-destination                                  -20 B   (-0.1%)       -1 B   (-0.0%)
async-load-layout-masterbar-drafts-popover                                 -20 B   (-0.0%)       +2 B   (+0.0%)
async-load-layout-guided-tours-component                                   -20 B   (-0.0%)       -1 B   (-0.0%)
async-load-calypso-reader-tag-stream-main                                  -20 B   (-0.1%)       -1 B   (-0.0%)
async-load-calypso-reader-list-manage                                      -20 B   (-0.1%)       +2 B   (+0.0%)
async-load-calypso-my-sites-current-site-stale-cart-items-notice           -20 B   (-0.0%)      -12 B   (-0.0%)
async-load-calypso-my-sites-current-site-domain-warnings                   -20 B   (-0.0%)       -2 B   (-0.0%)
async-load-calypso-layout-community-translator-launcher                    -20 B   (-0.1%)       -3 B   (-0.0%)
async-load-calypso-components-jetpack-sidebar                              -20 B   (-0.1%)       -4 B   (-0.1%)
async-load-calypso-blocks-calendar-popover                                 -20 B   (-0.0%)       -4 B   (-0.0%)
async-load-signup-steps-survey                                             -15 B   (-0.1%)       +1 B   (+0.0%)
async-load-signup-steps-site-style                                         -15 B   (-0.1%)       +1 B   (+0.0%)
async-load-signup-steps-rewind-migrate                                     -15 B   (-0.1%)       -3 B   (-0.0%)
async-load-signup-steps-rebrand-cities-welcome                             -15 B   (-0.1%)       +2 B   (+0.0%)
async-load-signup-steps-passwordless                                       -15 B   (-0.1%)       +1 B   (+0.0%)
async-load-signup-steps-p2-details                                         -15 B   (-0.2%)       +0 B
async-load-signup-steps-creds-permission                                   -15 B   (-0.0%)       +0 B
async-load-signup-steps-creds-confirm                                      -15 B   (-0.0%)       +0 B
async-load-signup-steps-clone-ready                                        -15 B   (-0.1%)       -2 B   (-0.0%)
async-load-calypso-layout-guided-tours                                     -15 B   (-0.1%)       -5 B   (-0.2%)
async-load-calypso-blocks-product-purchase-features-list                   -15 B   (-0.0%)       +1 B   (+0.0%)

React components that are loaded lazily, when a certain part of UI is displayed for the first time.

Legend

What is parsed and gzip size?

Parsed Size: Uncompressed size of the JS and CSS files. This much code needs to be parsed and stored in memory.
Gzip Size: Compressed size of the JS and CSS files. This much data needs to be downloaded over network.

Generated by performance advisor bot at iscalypsofastyet.com.

scinos · 2020-08-12T20:52:17Z

This PR is blocked because Jest doesn't know how to resolve packages with a module field. Jest provides way to write our own resolver, but its cache won't be clean up on watch mode (which may cause some confusion). See jestjs/jest#2702 (comment) for more details.

I raised a PR with the changes required to make the custom resolver work and it just got accepted! As soon as that change is released and we bump Jest to the new version this PR will become unblocked.

jsnajdr · 2020-08-17T05:58:45Z

If we point the module field to src/index.js, as this patch does for many packages, it means that we start publishing untranspiled sources on NPM. Was that really your intent?

Normally, the CJS and ESM builds differ only in the module format: one uses CJS exports, the other uses the import and export keywords. The rest of the code is still always transpiled to a widely compatible ES5 syntax, both for CJS and ESM.

After this patch, the code shipped with CJS and ESM will be different.

If we are OK with publishing untranspiled code to NPM, we can remove the package transpilation stuff completely, even for CJS. It's then up to the package consumer to figure out how to consume the sources.

scinos · 2020-08-17T06:03:50Z

If we point the module field to src/index.js, as this patch does for many packages, it means that we start publishing untranspiled sources on NPM. Was that really your intent?

IMO that's a good thing. We continue to publish a ES5 version, packed as CJS in main. This is how consumers can use our code in a way that JustWorks™️. We also publish a modern, untranspiled ES2020 version packed as ESM in module for advanced users.

I think that has more value than shipping ES5 in CJS and ESM.

scinos · 2020-08-17T06:20:05Z

An alternative would be to introduce a third property (esnext?) that points to the original source. We would have main with ES5+CJS, module with ES5+ESM, esnext with ES2020+ESM.

jsnajdr · 2020-08-17T06:21:28Z

An alternative would be to introduce a third property (esnext?)

Yeah, that's calypso:main, only with a different name.

jsnajdr · 2020-08-17T06:55:50Z

We continue to publish a ES5 version, packed as CJS in main. This is how consumers can use our code in a way that JustWorks.

That sounds good only until one starts to look more deeply at who these CJS consumers are and how they consume the package in practice.

If the consumer uses webpack, then webpack always looks at module field before main. For these consumers, the CJS version is practically invisible. The consumer will always use the module thing, which is newly untranspiled.

The change from transpiled to untranspiled requires caution on the consumer side. E.g., Jetpack uses i18n-calypso and it doesn't transpile it (that's specified in the webpack config). Just switching to the newly published version would lead to bundling untranspiled code into the output, including JSX, which would break a lot of things.

The only place where I can see the CJS version really used is require( 'a8c-package' ) in Node.js. But that's already kind of legacy -- if I write import from 'a8c-package' in latest Node.js, I'll get the untranspiled module pointed to by the module field, right? And that code will often contain JSX, causing a syntax error.

To summarize, I think there is a mismatch between what our packages expose and how the consumers actually consume it. That mismatch exists even without the changes in this PR -- the PR only highlights them.

scinos · 2020-08-17T07:09:37Z

If the consumer uses webpack, then webpack always looks at module field before main. For these consumers, the CJS version is practically invisible.

That's the default behaviour, but I'd say that's the consumer's responsibility to change if they are not ok with it. I do agree most people won't change it so for them, module is the "main" way to consume the package.

Just switching to the newly published version would lead to bundling untranspiled code into the output, including JSX, which would break a lot of things.

For the public packages, we could bump the major version number and document the breaking change.

if I write import from 'a8c-package' in latest Node.js, I'll get the untranspiled module pointed to by the module field, right?

AFAIK, node always ignores the module field. You will get main or exports (if defined).

I think we have three options:

Ship module with untranspiled+ESM (aka this PR). It is arguable the simplest option but will require consumers to adapt their webpack settings to also transpile these packages.
Ship a new esnext, calypso:main or any other bespoke name. Adds the complexity of having to remember what that custom field means, but it won't require changes in the consumers if they don't want to. Only those who want to get the benefit of the untranspiled code (eg: Calypso) will have to change.
Go with the new conditional exports. It is more future-proof and the way we will eventually go with, but at the moment there is no support in Webpack 4 and no standard way of exposing untranspiled code (webpack proposed a browser field).

Sounds like (2) is the most sensible approach right now. We just need a name. esnext may get confused with the old esnext (https://2ality.com/2017/06/pkg-esnext.html), calypso:main is too coupled with Calypso IMO. What about just src? It kind of suggests that it points to the original source, not transpiled in any way (may include TS, JSX, ES.next features...).

jsnajdr · 2020-08-17T07:45:48Z

We can also specify the requirements by consumers and then work from there.

For browsers, I think the only practical consumer are bundlers, and to these, we can simply ship the untranspiled sources (even TypeScript) and shift the responsibility entirely to the consumer, i.e., the webpack config author. It's the consumer, not the package publisher, who knows what their target browsers and polyfills are. Transpilation by publishers only leads to suboptimal code (i.e., transpiled too much).

Then there's the Node.js runtime that imports the package at runtime using the Node.js resolver and loader. Node.js doesn't understand JSX, so some transpilation by publisher will always need to happen. (Here the typical package is a React component that's also used for SSR in Node.)

For Node.js without ESM and exports, we need a CJS export pointed to by main field, with transpiled JSX and reasonably stable JS syntax.

For latest Node.js with ESM unflagged, we need an ESM export with the same code content as CJS.

Can all that be encoded in package.json in such a way that each consumer picks up the correct thing? I don't know at this moment.

Go with the new conditional exports. It is more future-proof and the way we will eventually go with, but at the moment there is no support in Webpack 4

Is there support in Webpack 5? We can write a webpack plugin for exports, that shouldn't be hard.

no standard way of exposing untranspiled code

This standardization debate has been going on for years, with no results. I'm afraid we'll need to go with something that's not standard, but is reasonably well designed and has a chance for wider adoption.

calypso:main is too coupled with Calypso IMO. What about just src?

This article proposes browser2017.

jsnajdr · 2020-08-17T08:17:18Z

When exposing the different exports from a package, my experience from #43679 has been that the biggest bottleneck is the TypeScript compiler. It has its own resolver which is not configurable. It's possible to write your own resolver, but that involves implementing the "compiler host" interface and somehow wiring it all together.

It affects JS-only packages, too, as long as a TS source imports them. The TS compiler will look for the index.d.ts type definitions, the types field... TS needs to be treated as another common consumer that needs to get the right thing when importing a JS package.

scinos · 2020-08-17T08:23:54Z

calypso:main is too coupled with Calypso IMO. What about just src?

This article proposes browser2017.

The article also proposes esnext (equivalent to webpack's conditional browser proposal) as an alternative.
I don't think any of those applies, because those are for exposing transpiled code. I think we should introduce src as a way to expose untranspiled code

@jsnajdr In your opinion, is the name of the export the only thing that needs to be changed to get this merged to master? Or is there something else?

jsnajdr · 2020-08-17T08:32:30Z

In your opinion, is the name of the export the only thing that needs to be changed to get this merged to master?

I'd like to discuss and design the package-export strategy as a whole before starting to ship code. Right now, I'm working on understanding the conditional exports proposal more deeply. These are big changes and I think that trying to merge this or #43679 as expediently as possible is not the right focus.

jsnajdr · 2020-08-17T11:18:30Z

It seems that the current patch is able to satisfy the requirements in #44824 (comment):

The module field points to the untranspiled sources, and it is what is picked by bundlers to further transpile and process. Node.js reportedly never looks at the module field: it only looks at main or the new exports.

When loaded natively by Node.js, the dist/cjs folder pointed to by main field is used. That makes the package consumable by Node.js apps.

No custom field (esnext or calypso:main) is strictly needed here.

Open questions:

the dist/esm folder seems to be redundant now. There is no use case for it.
can TypeScript compiler always load the affected packages' types?
there are packages/subpackages that export different files for browsers and for Node.js, using the browser field. Are we compatible with that?

jsnajdr · 2020-10-22T08:58:47Z

@scinos How do you think this PR is related to #43679 which has a similar goal? They both now use calypso:main or calypso:src. Is there a big overlap between them? Are they at odds with each other?

I didn't look at this problem in the last few months, so I forgot almost everything I knew about both PRs 🙂

scinos · 2020-10-22T09:03:08Z

I think this PR is a subset of #43679. #43679 also changes TypeScript projects and bundles all node_modules into server bundler.

jsnajdr · 2020-10-22T09:44:40Z

bundles all node_modules into server bundler.

Yeah, that's superseded by the packagesInMonorepo test, also present in #45774 in a slightly different form.

Thanks for reviving this effort -- it's an important improvement that's been dormant for a few months unfortunately 🙁

scinos · 2020-10-22T10:17:20Z

Strange, I expected all bundles to be smaller as they are transpiled for evergreen now, but somehow some sections got bigger 🤨 . I'll investigate

scinos · 2020-10-22T13:03:19Z

Strange, I expected all bundles to be smaller as they are transpiled for evergreen now

False alarm. Changes in chunk sizes are caused by webpack heuristics moving things around, but overall, this PR makes evergreen build ~30kb smaller (parsed size, from http://iscalypsofastyet.com/p/branch?branch=try/dont-transpile-packages). Manually inspecting a few compiled files I can see the diff comes, as expected, to a more effective transpilation (eg: there are no more _babel_runtime_helpers_objectSpread2 helpers)

sirbrillig · 2020-10-22T20:27:38Z

Just FYI that I created a new package, shopping-cart, in #46532, in case that also needs to be updated here.

scinos · 2020-10-23T04:15:35Z

@sirbrillig Thank you for letting me know :) I'll check it out and include it in this PR if needed

jsnajdr

@scinos would you mind if I pushed a few little changes to this branch? Locally, I did a rebase to latest master and removed the packagesInMonorepo code, as the same thing has been merged this week together with #45774. I guess that I could just push it when I already did the work locally, in order to test the PR.

To confirm my understanding: this PR adds calypso:src and skips build for packages that are written in standard JS and transpiled using Babel. It keeps TypeScript packages unchanged, mainly because of the troubles with configuring a custom resolver for TypeScript compiler. In #43679 I tried to remove the build step for TypeScript packages, too, but that works only when they are compiled with Babel (with TS plugin). tsc doesn't find the imported modules.

jsnajdr · 2020-10-30T09:10:53Z

packages/babel-plugin-transform-wpcalypso-async/jest.config.js

@@ -2,4 +2,5 @@ module.exports = {
 	preset: '@automattic/calypso-build',
 	rootDir: __dirname,
 	cacheDirectory: '<rootDir>/../../.cache/jest',
+	resolver: '<rootDir>/../../test/module-resolver.js',


This would be nice to move to some shared preset. The packages' Jest configs are getting repetitive: the cacheDirectory is always the same, and now the resolver field is, too.

By the way, even the rootDir field could be removed one day, when this Jest bug gets fixed.

I couldn't figure out how to do it. When I added the cacheDirectory to the base jest config I got some problems relative to how jest resolves paths... I don't remember the details, I'll try again.

Created a separate PR for it: #46953

jsnajdr · 2020-10-30T09:15:28Z

test/module-resolver.js

+		packageFilter: ( pkg ) => {
+			//TODO: when Jest moves to resolver v2, this method will receive a second argument that points to the real package file (ie: no symlink)
+			//We can use it to determine if the package file is within the repo
+			if ( packages.includes( pkg.name ) ) {


Is this filtering by name even needed? What if we just checked if calypso:src field is present in package.json and returned a patched main if it is?

Good point, probably not needed, I'll remove it.

scinos · 2020-11-03T05:18:55Z

@scinos would you mind if I pushed a few little changes to this branch?

Of course I don't mind! :) Feel free to improve it.

did a rebase to latest master and removed the packagesInMonorepo code

I've made those changes as well.

To confirm my understanding: this PR adds calypso:src and skips build for packages that are written in standard JS and transpiled using Babel. It keeps TypeScript packages unchanged, mainly because of the troubles with configuring a custom resolver for TypeScript compiler

Correct.

jsnajdr · 2020-11-03T07:32:19Z

I've made those changes as well.

Yeah, you already made the changes I had in mind 🙂 I have nothing further to add.

jsnajdr

Looks good, including the Jest config sub-PR.

In a related PR, I'm trying to reduce the TypeScript busywork in #47002.

scinos self-assigned this Aug 11, 2020

scinos added the [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. label Aug 11, 2020

scinos added the [Status] In Progress label Aug 11, 2020

scinos marked this pull request as draft August 11, 2020 13:02

scinos added the [Status] Blocked / Hold label Aug 11, 2020

scinos force-pushed the try/dont-transpile-packages branch from 9d76bd8 to 7ab4cea Compare August 12, 2020 14:26

scinos force-pushed the try/dont-transpile-packages branch from 7ab4cea to 9680f61 Compare August 13, 2020 09:25

scinos removed the [Status] Blocked / Hold label Aug 13, 2020

scinos requested a review from a team August 17, 2020 05:38

scinos removed the [Status] In Progress label Aug 17, 2020

scinos requested review from jsnajdr and a team and removed request for a team August 17, 2020 05:38

scinos mentioned this pull request Aug 17, 2020

Consume monorepo packages without building them #43679

Draft

scinos marked this pull request as ready for review August 17, 2020 05:41

scinos added the [Status] Blocked / Hold label Aug 17, 2020

scinos requested a review from a team October 22, 2020 12:40

scinos requested a review from jsnajdr October 22, 2020 13:03

scinos removed the [Status] In Progress label Oct 22, 2020

scinos force-pushed the try/dont-transpile-packages branch 3 times, most recently from cc653e8 to e00b567 Compare October 27, 2020 08:53

jsnajdr reviewed Oct 30, 2020

View reviewed changes

scinos added 4 commits October 30, 2020 12:17

Re-introduce calypso:main

ec334fd

Adds missing resolver to jest config

c4916c0

Change composite-checkout's webpack config to support calypso:src

04ada0c

Bundle packages in the monorepo to

8787b52

scinos force-pushed the try/dont-transpile-packages branch from e00b567 to 8787b52 Compare October 30, 2020 13:50

scinos added 2 commits October 30, 2020 14:53

Simplify resolver

cbf3037

Delete redundant packagesInMonorepo

88b0da9

jsnajdr mentioned this pull request Nov 2, 2020

Optimize how TypeScript-authored packages are built #47002

Merged

jsnajdr approved these changes Nov 3, 2020

View reviewed changes

scinos merged commit b20ddce into master Nov 3, 2020

scinos deleted the try/dont-transpile-packages branch November 3, 2020 08:13

matticbot removed the [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. label Nov 3, 2020

sirbrillig mentioned this pull request Nov 3, 2020

Checkout: Update npm package configs to match other packages #47070

Closed

scinos mentioned this pull request Nov 17, 2020

Use babel to transpile TS projects #47502

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Do not transpile JS packages twice #44824

Do not transpile JS packages twice #44824

scinos commented Aug 11, 2020 •

edited

Loading

matticbot commented Aug 11, 2020

matticbot commented Aug 11, 2020 •

edited

Loading

scinos commented Aug 12, 2020 •

edited

Loading

jsnajdr commented Aug 17, 2020

scinos commented Aug 17, 2020 •

edited

Loading

scinos commented Aug 17, 2020

jsnajdr commented Aug 17, 2020

jsnajdr commented Aug 17, 2020

scinos commented Aug 17, 2020

jsnajdr commented Aug 17, 2020

jsnajdr commented Aug 17, 2020

scinos commented Aug 17, 2020

jsnajdr commented Aug 17, 2020

jsnajdr commented Aug 17, 2020

jsnajdr commented Oct 22, 2020

scinos commented Oct 22, 2020

jsnajdr commented Oct 22, 2020

scinos commented Oct 22, 2020

scinos commented Oct 22, 2020

sirbrillig commented Oct 22, 2020

scinos commented Oct 23, 2020 •

edited

Loading

jsnajdr left a comment

jsnajdr Oct 30, 2020

scinos Oct 30, 2020

scinos Oct 30, 2020 •

edited

Loading

jsnajdr Oct 30, 2020

scinos Oct 30, 2020

scinos commented Nov 3, 2020 •

edited

Loading

jsnajdr commented Nov 3, 2020

jsnajdr left a comment

Do not transpile JS packages twice #44824

Do not transpile JS packages twice #44824

Conversation

scinos commented Aug 11, 2020 • edited Loading

Summary

Background

Changes

Testing instructions

Update (22/Oct)

matticbot commented Aug 11, 2020

matticbot commented Aug 11, 2020 • edited Loading

scinos commented Aug 12, 2020 • edited Loading

jsnajdr commented Aug 17, 2020

scinos commented Aug 17, 2020 • edited Loading

scinos commented Aug 17, 2020

jsnajdr commented Aug 17, 2020

jsnajdr commented Aug 17, 2020

scinos commented Aug 17, 2020

jsnajdr commented Aug 17, 2020

jsnajdr commented Aug 17, 2020

scinos commented Aug 17, 2020

jsnajdr commented Aug 17, 2020

jsnajdr commented Aug 17, 2020

jsnajdr commented Oct 22, 2020

scinos commented Oct 22, 2020

jsnajdr commented Oct 22, 2020

scinos commented Oct 22, 2020

scinos commented Oct 22, 2020

sirbrillig commented Oct 22, 2020

scinos commented Oct 23, 2020 • edited Loading

jsnajdr left a comment

Choose a reason for hiding this comment

jsnajdr Oct 30, 2020

Choose a reason for hiding this comment

scinos Oct 30, 2020

Choose a reason for hiding this comment

scinos Oct 30, 2020 • edited Loading

Choose a reason for hiding this comment

jsnajdr Oct 30, 2020

Choose a reason for hiding this comment

scinos Oct 30, 2020

Choose a reason for hiding this comment

scinos commented Nov 3, 2020 • edited Loading

jsnajdr commented Nov 3, 2020

jsnajdr left a comment

Choose a reason for hiding this comment

scinos commented Aug 11, 2020 •

edited

Loading

matticbot commented Aug 11, 2020 •

edited

Loading

scinos commented Aug 12, 2020 •

edited

Loading

scinos commented Aug 17, 2020 •

edited

Loading

scinos commented Oct 23, 2020 •

edited

Loading

scinos Oct 30, 2020 •

edited

Loading

scinos commented Nov 3, 2020 •

edited

Loading