Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make client/ directory (Calypso app) a Lerna package #38190

Merged
merged 20 commits into from
Jan 16, 2020

Conversation

jsnajdr
Copy link
Member

@jsnajdr jsnajdr commented Dec 5, 2019

Turns the Calypso client (and server) into proper Lerna monorepo package. The top-level package.json now contains only dev dependencies for the whole monorepo, and there is a new client/package.json that now has the runtime dependencies for Calypso client and server.

The main benefit is that we can now have a single unified node_modules tree, installed with one npm ci command, and with one global package-lock.json. Separate lockfiles and NPM installs in apps/ were removed. The only part of the monorepo that continues to have its own NPM tree is tests/e2e.

The server/ directory was moved to client/server/ to make the Calypso app self-contained inside one directory. Many import paths in the server/ sources needed to be updated and clarified.

The root package.json declares the app-like subpackages as dependencies to make them part of the npm i tree.

Some devDependencies were incorrectly declared as dependencies, so I moved them to the right place.

Some areas that are not 100%

  • in Dockerfile, I had to disable the separate layer for npm ci. If I understand the logic correctly, it first copies a minimal version of the source tree that's needed to run the npm install, then installs the node_modules, and only then copies and builds the rest? @sirreal it would be nice if we could make it work again
  • there are probably more eslint-plugin-import errors reported by ESLint now than needed. Can we improve on that?
  • the wp-desktop is failing. 🙂

How to test:

  • test the Full Site Editing and WP.com Block Editor apps -- do they build and run as expected? Cc @Automattic/cylon as I personally have only limited knowledge about these apps.
  • does Calypso really work 100%? The hardest part to make right were devdocs.

@matticbot
Copy link
Contributor

@jsnajdr jsnajdr force-pushed the move/client-to-package branch from 5e67e16 to 3f442d9 Compare December 5, 2019 10:32
"yargs": "13.3.0"
"@automattic/full-site-editing": "file:apps/full-site-editing",
"@automattic/wpcom-block-editor": "file:apps/wpcom-block-editor",
"wp-calypso": "file:client"
Copy link
Contributor

@blowery blowery Dec 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we go this route we should probably include everything in apps/*?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, and also all the packages in packages/ that are not used by Calypso or something else in the repo transitively.

In other words, all packages in the monorepo should be direct or indirect dependencies of the root package.json. That ensures that npm install installs the whole package tree for the whole monorepo.

Gutenberg does it: https://github.com/WordPress/gutenberg/blob/master/package.json#L20-L66

Alternatively, lerna bootstrap does the npm installation process, but bails out to a simple npm install if it sees a local file: ref in dependencies in the root package.json. We could also go that way, but leaving the npm tree solely to the npm client is preferrable.

Copy link
Contributor

@blowery blowery Dec 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively, lerna bootstrap does the npm installation process, but bails out to a simple npm install if it sees a local file: ref in dependencies in the root package.json. We could also go that way, but leaving the npm tree solely to the npm client is preferrable.

Yeah, if we want to use lerna bootstrap, we'd also want to jump back to semver ranges instead of file refs. Once you add one file ref, bootstrap stops doing anything useful.

If we really want a mix, something like yarn workspaces or pnpm workspaces offer a lot more flexibility.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way I've been thinking about it: lerna provides the "I want to run a command against the package graph" functionality and something else (npm, yarn, pnpm) provides the install and graph linkage. lerna bootstrap is to be avoided as it's just a hacky way of doing what one of the package managers would rather do itself.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Totally agreed that lerna bootstrap is to be avoided. Even the Lerna maintainer told us that it's a buggy and hard-to-maintain hack that tries to do the package manager's job (poorly).

@simison
Copy link
Member

simison commented Jan 9, 2020

cc @tyxla this PR might be relevant movement to follow regarding Jetpack Cloud living in /apps.

@jsnajdr jsnajdr force-pushed the move/client-to-package branch from 88ec205 to 4368ec1 Compare January 9, 2020 17:21
@matticbot
Copy link
Contributor

matticbot commented Jan 10, 2020

Here is how your PR affects size of JS and CSS bundles shipped to the user's browser:

App Entrypoints (~11720 bytes removed 📉 [gzipped])

name                   parsed_size           gzip_size
entry-gutenboarding       -25540 B  (-1.4%)    -6440 B  (-1.3%)
entry-main                 -8905 B  (-0.5%)    -2722 B  (-0.7%)
entry-login                -5372 B  (-0.5%)    -1744 B  (-0.6%)
entry-domains-landing       -152 B  (-0.0%)     -814 B  (-0.6%)

Common code that is always downloaded and parsed every time the app is loaded, no matter which route is used.

Sections (~48194 bytes removed 📉 [gzipped])

name                      parsed_size            gzip_size
theme                       -143500 B  (-36.3%)   -31099 B  (-31.7%)
post-editor                  -13204 B   (-0.7%)    -2022 B   (-0.4%)
woocommerce                   -8610 B   (-0.4%)    -1523 B   (-0.3%)
plugins                       -8559 B   (-1.8%)      -86 B   (-0.1%)
themes                        -7536 B   (-2.3%)     -133 B   (-0.2%)
domains                       -7279 B   (-0.9%)     -674 B   (-0.4%)
purchases                     -7259 B   (-0.8%)     -500 B   (-0.2%)
stats                         -7051 B   (-0.9%)     -488 B   (-0.3%)
hosting                       -6435 B   (-2.7%)     +104 B   (+0.2%)
settings                      -6433 B   (-1.3%)     -375 B   (-0.3%)
jetpack-connect               -6333 B   (-1.1%)     -606 B   (-0.4%)
checkout                      -5936 B   (-0.5%)     -638 B   (-0.2%)
plans                         -5729 B   (-1.3%)     -492 B   (-0.4%)
reader                        -5668 B   (-1.4%)     -413 B   (-0.4%)
gutenberg-editor              -5633 B   (-0.8%)     -296 B   (-0.2%)
marketing                     -5337 B   (-1.3%)     -343 B   (-0.3%)
media                         -5168 B   (-1.4%)     -340 B   (-0.4%)
settings-writing              -4715 B   (-1.0%)     -291 B   (-0.3%)
posts                         -4620 B   (-1.6%)     -290 B   (-0.4%)
home                          -4592 B   (-1.8%)     -335 B   (-0.5%)
posts-custom                  -4588 B   (-1.6%)     -282 B   (-0.4%)
activity                      -4574 B   (-1.0%)     -225 B   (-0.2%)
settings-security             -4286 B   (-1.6%)     -246 B   (-0.4%)
email                         -4100 B   (-1.4%)     -332 B   (-0.5%)
help                          -3996 B   (-0.9%)     -232 B   (-0.2%)
people                        -3757 B   (-1.1%)     -263 B   (-0.3%)
earn                          -3696 B   (-1.4%)     -300 B   (-0.5%)
security                      -3691 B   (-0.9%)     -244 B   (-0.2%)
checklist                     -3686 B   (-1.3%)     -227 B   (-0.3%)
settings-performance          -3456 B   (-1.7%)     -230 B   (-0.4%)
google-my-business            -3425 B   (-1.3%)     -448 B   (-0.6%)
pages                         -3320 B   (-1.4%)     -233 B   (-0.4%)
notification-settings         -3110 B   (-1.0%)     -239 B   (-0.3%)
concierge                     -2994 B   (-1.0%)     -297 B   (-0.4%)
account                       -2992 B   (-1.0%)     -167 B   (-0.2%)
settings-discussion           -2937 B   (-1.7%)     -191 B   (-0.4%)
comments                      -2936 B   (-0.6%)     -241 B   (-0.2%)
account-close                 -2805 B   (-1.0%)     -154 B   (-0.2%)
import                        -2668 B   (-1.3%)     -270 B   (-0.5%)
wp-super-cache                -2628 B   (-1.2%)     -180 B   (-0.3%)
signup                        -2608 B   (-1.6%)     -174 B   (-0.4%)
customize                     -2576 B   (-1.5%)     -206 B   (-0.4%)
export                        -2523 B   (-1.5%)     -165 B   (-0.4%)
devdocs                       -2363 B   (-1.5%)      -40 B   (-0.1%)
happychat                     -2352 B   (-1.0%)      -98 B   (-0.1%)
zoninator                     -2298 B   (-0.9%)     -227 B   (-0.3%)
feature-upsell                -2267 B   (-1.6%)     -137 B   (-0.4%)
site-blocks                   -2213 B   (-0.9%)     -113 B   (-0.2%)
privacy                       -2153 B   (-1.0%)      -97 B   (-0.2%)
me                            -2024 B   (-0.9%)      -82 B   (-0.1%)
migrate                       -1964 B   (-1.8%)     -167 B   (-0.6%)
preview                       -1679 B   (-1.8%)     -134 B   (-0.5%)
sensei                        -1652 B   (-1.8%)     -171 B   (-0.7%)
hello-dolly                   -1645 B   (-1.7%)     -152 B   (-0.6%)
sites                         -1594 B   (-1.8%)     -141 B   (-0.6%)
accept-invite                 -1247 B   (-0.7%)     -106 B   (-0.2%)
auth                           -485 B   (-2.6%)      -78 B   (-1.5%)
domain-connect-authorize       -344 B   (-2.7%)      -60 B   (-1.6%)
mailing-lists                   -49 B   (-0.7%)       -5 B   (-0.3%)

Sections contain code specific for a given set of routes. Is downloaded and parsed only when a particular route is navigated to.

Async-loaded Components (~5832 bytes removed 📉 [gzipped])

name                                                         parsed_size           gzip_size
async-load-design-blocks                                        -15117 B  (-0.6%)     -821 B  (-0.1%)
async-load-design                                                -5373 B  (-0.3%)     -919 B  (-0.2%)
async-load-design-playground                                     -4442 B  (-0.3%)     -959 B  (-0.2%)
async-load-components-web-preview-component                      -3928 B  (-1.1%)     -211 B  (-0.2%)
async-load-post-editor-media-modal                               -3207 B  (-1.2%)     -203 B  (-0.3%)
async-load-lib-happychat-connection                              -2656 B  (-4.1%)      -42 B  (-0.2%)
async-load-blocks-inline-help-popover                            -2624 B  (-1.1%)     -112 B  (-0.2%)
async-load-signup-steps-plans                                    -2240 B  (-1.4%)     -160 B  (-0.4%)
async-load-signup-steps-domains                                  -2144 B  (-1.0%)     -218 B  (-0.4%)
async-load-signup-steps-clone-point                              -1089 B  (-0.7%)      -40 B  (-0.1%)
async-load-signup-steps-user                                     -1069 B  (-0.9%)      -30 B  (-0.1%)
async-load-signup-steps-plans-atomic-store                       -1053 B  (-1.1%)     -108 B  (-0.4%)
async-load-reader-sidebar                                         -979 B  (-1.8%)      -84 B  (-0.6%)
async-load-my-sites-checklist-wpcom-checklist-component-jsx       -926 B  (-0.7%)     -167 B  (-0.5%)
async-load-reader-following-manage                                -910 B  (-0.7%)      -89 B  (-0.3%)
async-load-layout-guided-tours                                    -905 B  (-2.4%)       -7 B  (-0.1%)
async-load-extensions-woocommerce-app-store-stats                 -802 B  (-1.8%)      -39 B  (-0.3%)
async-load-blocks-product-purchase-features-list                  -790 B  (-1.8%)      -56 B  (-0.5%)
async-load-signup-steps-clone-credentials                         -739 B  (-1.6%)      -35 B  (-0.3%)
async-load-extensions-woocommerce-app-store-stats-referrers       -661 B  (-1.5%)      -53 B  (-0.4%)
async-load-layout-guided-tours-component                          -654 B  (-0.7%)      -32 B  (-0.2%)
async-load-reader-search-stream                                   -641 B  (-0.6%)      -75 B  (-0.3%)
async-load-signup-steps-rewind-form-creds                         -638 B  (-1.6%)      -23 B  (-0.2%)
async-load-my-sites-current-site-notice                           -605 B  (-1.8%)      -33 B  (-0.4%)
async-load-blocks-reader-full-post                                -599 B  (-1.4%)      -24 B  (-0.2%)
async-load-reader-feed-stream                                     -586 B  (-2.3%)      -49 B  (-0.7%)
async-load-reader-site-stream                                     -585 B  (-2.3%)      -48 B  (-0.7%)
async-load-my-sites-site-settings-seo-settings-form               +523 B  (+0.2%)      +67 B  (+0.1%)
async-load-apps-notifications-index-jsx                           -515 B  (-0.4%)     -173 B  (-0.5%)
async-load-signup-steps-about                                     -473 B  (-1.0%)      -40 B  (-0.3%)
async-load-blocks-support-article-dialog-dialog                   -431 B  (-0.8%)      -21 B  (-0.1%)
async-load-blocks-inline-help                                     -428 B  (-0.7%)      -19 B  (-0.1%)
async-load-post-editor-editor-sharing-accordion                   -427 B  (-1.9%)      -15 B  (-0.3%)
async-load-signup-steps-creds-confirm                             -394 B  (-1.7%)      -12 B  (-0.2%)
async-load-signup-steps-creds-permission                          -375 B  (-1.6%)      -17 B  (-0.3%)
async-load-my-sites-guided-transfer                               -365 B  (-1.5%)      -32 B  (-0.6%)
async-load-extensions-woocommerce-app-store-stats-listview        -357 B  (-2.2%)      -37 B  (-0.8%)
async-load-components-happychat                                   -355 B  (-1.0%)      -12 B  (-0.1%)
async-load-signup-steps-theme-selection                           -326 B  (-0.8%)      -41 B  (-0.4%)
async-load-my-sites-current-site-stale-cart-items-notice          -325 B  (-0.8%)      -22 B  (-0.2%)
async-load-signup-steps-site                                      -318 B  (-1.3%)      -24 B  (-0.4%)
async-load-signup-steps-rewind-migrate                            -313 B  (-1.3%)      -14 B  (-0.2%)
async-load-quick-language-switcher                                -304 B  (-1.2%)      -23 B  (-0.3%)
async-load-signup-steps-passwordless                              -292 B  (-1.2%)      -17 B  (-0.3%)
async-load-signup-steps-site-topic                                -291 B  (-1.0%)      -41 B  (-0.5%)
async-load-signup-steps-clone-ready                               -285 B  (-1.0%)      -20 B  (-0.3%)
async-load-signup-steps-import-url-onboarding                     -283 B  (-0.9%)      -23 B  (-0.3%)
async-load-reader-tag-stream-main                                 -269 B  (-1.7%)      -43 B  (-0.9%)
async-load-post-editor-editor-author                              -259 B  (-1.4%)      -17 B  (-0.3%)
async-load-blocks-jitm-templates-default                          -248 B  (-2.2%)      -13 B  (-0.4%)
async-load-my-sites-current-site-domain-warnings                  -231 B  (-0.4%)      -49 B  (-0.4%)
async-load-signup-steps-import-url                                -227 B  (-1.1%)      -47 B  (-0.8%)
async-load-layout-nps-survey-notice                               -222 B  (-1.0%)      -10 B  (-0.2%)
async-load-lib-preferences-helper                                 -212 B  (-1.4%)     -144 B  (-3.0%)
async-load-signup-steps-site-type                                 -203 B  (-1.7%)      -15 B  (-0.5%)
async-load-signup-steps-rebrand-cities-welcome                    -200 B  (-1.5%)      -11 B  (-0.3%)
async-load-signup-steps-survey                                    -188 B  (-1.1%)      -16 B  (-0.4%)
async-load-signup-steps-site-title                                -188 B  (-1.6%)       -6 B  (-0.2%)
async-load-signup-steps-site-style                                -188 B  (-0.6%)      -15 B  (-0.2%)
async-load-signup-steps-clone-destination                         -187 B  (-1.0%)      -18 B  (-0.4%)
async-load-signup-steps-site-or-domain                            -181 B  (-0.9%)      -27 B  (-0.5%)
async-load-layout-community-translator-launcher                   -180 B  (-1.5%)      -14 B  (-0.3%)
async-load-signup-steps-reader-landing                            -177 B  (-1.5%)      -12 B  (-0.4%)
async-load-signup-steps-import-preview                            -159 B  (-0.7%)      -13 B  (-0.2%)
async-load-post-editor-editor-location                            -159 B  (-1.6%)      -16 B  (-0.5%)
async-load-blocks-app-banner                                      -155 B  (-1.8%)      -23 B  (-0.8%)
async-load-blocks-product-plan-overlap-notices                    -153 B  (-2.4%)      -14 B  (-0.7%)
async-load-blocks-calendar-popover                                +149 B  (+0.1%)      +35 B  (+0.1%)
async-load-signup-steps-clone-jetpack                             -148 B  (-1.4%)       -9 B  (-0.3%)
async-load-signup-steps-site-picker                               -145 B  (-1.6%)      -11 B  (-0.4%)
async-load-layout-masterbar-drafts-popover                        -145 B  (-1.9%)      -13 B  (-0.5%)
async-load-post-editor-editor-post-formats-accordion              -142 B  (-2.7%)      -10 B  (-0.6%)
async-load-reader-list-stream                                     -140 B  (-1.3%)      -15 B  (-0.5%)
async-load-blocks-jitm                                            -134 B  (-2.7%)       -9 B  (-0.5%)
async-load-post-editor-editor-seo-accordion                       -124 B  (-2.1%)       -4 B  (-0.2%)
async-load-signup-steps-clone-start                               -119 B  (-0.8%)      -13 B  (-0.3%)
async-load-signup-steps-rewind-were-backing                       -112 B  (-1.3%)       -9 B  (-0.3%)
async-load-signup-steps-creds-complete                            -105 B  (-1.2%)       -6 B  (-0.2%)
async-load-signup-steps-clone-cloning                             -104 B  (-0.8%)       -9 B  (-0.2%)
async-load-signup-steps-test-step                                 -102 B  (-1.3%)       -5 B  (-0.2%)
async-load-reader-conversations-stream                             -99 B  (-2.1%)       -3 B  (-0.2%)
async-load-post-editor-editor-discussion                           -99 B  (-2.0%)       -4 B  (-0.3%)
async-load-featured-image                                          -88 B  (-3.0%)       -9 B  (-0.9%)
async-load-blocks-login-social-connect-prompt                      -56 B  (-1.4%)       -4 B  (-0.3%)
async-load-lib-network-connection                                  -54 B  (-1.3%)      +12 B  (+0.8%)
async-load-lib-abtest-test-helper                                  -54 B  (-0.6%)       -5 B  (-0.2%)
async-load-components-webpack-build-monitor                        -39 B  (-1.5%)       -2 B  (-0.2%)
async-load-docs-selectors                                          -36 B  (-0.6%)       -4 B  (-0.2%)
async-load-components-signup-site-preview-component                -30 B  (-0.4%)       -4 B  (-0.2%)
async-load-blocks-jitm-templates-sidebar-banner                    -27 B  (-2.6%)       -8 B  (-1.8%)
async-load-blocks-inline-help-dialog                               -23 B  (-0.5%)       -6 B  (-0.3%)
async-load-blocks-jitm-templates-notice                            -20 B  (-3.0%)       -7 B  (-1.7%)
async-load-design-typography                                       -18 B  (-0.3%)       -4 B  (-0.2%)
async-load-signup-steps-launch-site                                -12 B  (-2.2%)       -5 B  (-1.5%)
async-load-reader-team-main                                        -12 B  (-3.1%)       -5 B  (-1.9%)

React components that are loaded lazily, when a certain part of UI is displayed for the first time.

Legend

What is parsed and gzip size?

Parsed Size: Uncompressed size of the JS and CSS files. This much code needs to be parsed and stored in memory.
Gzip Size: Compressed size of the JS and CSS files. This much data needs to be downloaded over network.

Generated by performance advisor bot at iscalypsofastyet.com.

@jsnajdr jsnajdr self-assigned this Jan 10, 2020
@jsnajdr jsnajdr added [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. and removed [Status] In Progress labels Jan 10, 2020
@jsnajdr jsnajdr force-pushed the move/client-to-package branch from ae35311 to 5d707df Compare January 10, 2020 09:51
@jsnajdr
Copy link
Member Author

jsnajdr commented Jan 10, 2020

This seems to work pretty well at this moment and I think we are ready for review and testing 😊

@ockham
Copy link
Contributor

ockham commented Jan 10, 2020

Would we consider (possibly as a follow-up) git mv client/ apps/calypso/?

@noahtallen
Copy link
Contributor

@jsnajdr are there particular changes which could affect FSE here? (i.e. any idea what I should look out for when testing?) I see that the package-lock files for the individual apps are removed, which is nice :)

The request string and the externals declaration must exactly match. That makes
sure that `search-index.js` is not bundled into server bundle, but continues to be
read with `require( 'server/bundler/search-index')`.

Also removes legacy externals like `hot-reloader` or `bundler/assets` (the assets file
is read with `fs.readFile`, not with Node require)
Don't specify a `packageDir` and make it look at the closest `package.json` by default.

Also, add `lodash` as a top-level dev dependency, as some build-only files (e.g., `.eslintrc`)
use it.
Allow `devDependencies` only in the root `package.json` and not anywhere else.
@jsnajdr jsnajdr force-pushed the move/client-to-package branch from 5db1083 to 9e9de37 Compare January 16, 2020 09:11
@jsnajdr jsnajdr merged commit 8e08325 into master Jan 16, 2020
@jsnajdr jsnajdr deleted the move/client-to-package branch January 16, 2020 12:21
@matticbot matticbot removed the [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. label Jan 16, 2020
@ockham
Copy link
Contributor

ockham commented Jan 16, 2020

👋 @jsnajdr I noticed that this PR moved the root Webpack configs to client/, but not the Babel and PostCSS ones. Shouldn't those also be moved there? They're only for Calypso-the-app, aren't they?

@jsnajdr
Copy link
Member Author

jsnajdr commented Jan 17, 2020

I noticed that this PR moved the root Webpack configs to client/, but not the Babel and PostCSS ones. Shouldn't those also be moved there?

That's a good question! When I tried to move babel.config.js to client/, I couldn't make Jest work. The babel-jest plugin is a zero-config thing and it couldn't find the Babel config in client/.

The babel.config.js file is supposed to live in the project root, or more precisely, in the cwd of the Node process that runs the build scripts. And Calypso build scripts still live in the root package.json and are ran from the repo's root directory.

A possible solution is to never allow the babel.config.js lookup algorithm to run and always specify the config explicitly as JS object. We already do that in the webpack configs, but there are places like babel-jest that need to find the file.

I don't know if the PostCSS file can be safely moved. I forgot about it when working on the patch, as the CSS build was never broken because of it 🙂

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants