deps: Experiments with Corepack

[arcanis]

Important note: this PR doesn't replace npm by Yarn! It's isn't our intent! We're experimenting with providing both npm and Yarn (and pnpm), so that users can easily contribute to JS projects regardless of the package manager expected.

Context (note that pmm got renamed as Corepack in order to be an available package name on the npm registry):

• [Feature] Discussion around default package manager choice #15244
• Package manager manager ... discussion TSC#904
• In short, the intent is to provide standard shims that allow users to run Yarn and pnpm commands without having to explicitly install them first, and without cluttering the Node distribution. Additionally, it will also allow users to pin their package manager version to their projects, thus preventing issues that could arise due to backward incompatible behaviors (be it new bugs, or on the contrary fixed bugs they relied upon).

This PR adds Corepack into the Node distribution pipeline (only for Linux / OSX - I didn't find how the Windows build works). By itself, it only involves very few changes:

• It adds a new folder to the repository, deps/corepack (it only takes 300KB)
• Similar to deps/npm, this folder is mirrored into lib/node_modules/corepack
• Binaries are also symlinked into bin (using the same pipeline as npm)
• A script (make corepack-update) updates the deps/corepack content to match the repository main branch

As a result:

• Once this PR is applied, Yarn and pnpm are applied via the Corepack shims (documentation here)
• Users can still opt-out by running npm install -g yarn, which will overwrite the Corepack binary starting from 1.22.10

Questions:

• Should I add Windows support?
• Should I transfer the Corepack repository to the Node org? If so, how?
• What testing strategy do you recommend? Corepack itself is tested on its repository: here

Documentation:

I haven't written them yet, but I suspect I'll need to update the following pages:

• Dependency list to add a reference to Corepack
• API Index to copy/port the Corepack readme

Do you see additional locations I should be mindful of?

Checklist

• make -j4 test (UNIX), or vcbuild test (Windows) passes
• tests and/or benchmarks are included
• documentation is changed or added
• commit message follows commit guidelines

[jasnell]

Fantastic to see progress on this!

With regards to the questions:

Should I add Windows support?

I'd say that's a requirement, yes.

Should I transfer the Corepack repository to the Node org? If so, how?

To do so, we'd need an issue opened in the nodejs/admin repo proposing the change. Assuming there are no objections after a few days, it's considered accepted and we just move it in. There's no real fanfare here.

What testing strategy do you recommend? Corepack itself is tested on its repository: here

Take a look at the test/parallel/test-npm-* tests and follow a similar pattern. The testing does not have to be comprehensive but should be enough to be reasonably certain that changes in core haven't broken something in corepack.

[devsnek]

these files could be symlinks to a helper that reads process.execName or whatever it's called. it's a common pattern in gnu core utils so it shouldn't break anything.

[devsnek]

it also occurs to me that if someone does not want a specific package manager on their system (for example, i do not wish to have yarn on my system), this could be considered somewhat invasive. Are we planning to ship tars with and without corepack?

[jasnell]

@devsnek ... this would not install the binaries by default. It installs jumpers that would download the binaries on demand if needed and it would be possible to block it via configuration.

[devsnek]

I realize it doesn't come with the binaries, but it puts things in my path that installs them. Let's say I'd rather not have any of that. Would I have to bundle my own releases of node?

[zkochan]

What if it will ask you before downloading Yarn or pnpm?

Like "Do you want Yarn v2 be downloaded to your system? Y/n"

[jasnell]

I'm not sure if it's implemented yet or not, but there was also discussion around being able to set configuration options to disable the download. It would likely be possible to have the tool remove the jumpers from the path when requested to do so. For interactive installers, it should be possible to make that configurable (e.g. by default install all jumpers but give the user the option to select them during the install)

[arcanis]

Downloads can be disabled by setting COREPACK_ENABLE_NETWORK=0 in the environment (nodejs/corepack#4). Removing the jumpers isn't possible yet as I wasn't sure of the use case, although it would be possible if needed.

[ljharb]

@merceyz many tools are invoked directly, and not as part of a package manager - how would these tools detect it?

[arcanis]

I believe it should be up to these projects to make decisions on a per-case basis, as the semantic compatible path isn't clear. For instance, that CRA "will use Yarn to install dependencies (when available)" could be interpreted as CRA preferring installs to be made via Yarn as much as possible - in which case using npm when Yarn is available wouldn't be the behavior that the project would prefer.

[arcanis]

Also note that these tools are typically called via npx, pnpx, or yarn create / yarn dlx. In all of these cases, the tool got an implicit insight as to the preferred package manager (ie under npx the $npm_config_user_agent value will reference npm, with yarn dlx it'll reference Yarn, etc).

[bnb]

There still seem to be some lingering TODOs from the issues linked in the original point before this should land:

• what are the criteria for adding package managers to corepack
• what are the criteria for removing package mangers to corepack
• what are the criteria for evolutions of package managers (like Berry) being added to corepack
• do all the current package managers meet all those criteria
• are there any package managers that are left out despite meeting that criteria
• have we measured the impact on our consumers, like FaaS platforms or end-user developers
• what is that impact on our users, beyond speculation
• can users turn this off

I'd also like to ask:

• are we comfortable adding something this fundamental that has a single maintainer
• are we as the Node.js willing and 100% able to commit, without question, to supporting this for the next decade
• if we land this, shouldn't it be behind a flag

Additionally, I think it's worth being upfront about our intents here: do we plan to eventually try to install npm through this? This should be probably be discussed since it could be impactful in deciding how to approach landing this / if it should be landed. Others shared their views in some of the aforementioned issues, and I have my own opinions on this that I'd be more than happy to share if they are potentially relevant.

[mmarchini]

marking objection to land explicitly until we:

• move the project to the org
• (optional) onboard more maintainers (as per @bnb comment)
• make sure this ships as experimental somehow (so it can be included in v15)
  • I don't think a doc-only experimental notice is enough, this should probably be an opt-in feature on v15

[mmarchini]

Should I add Windows support?

All tier 1 and tier 2 platforms listed on https://github.com/nodejs/node/blob/master/BUILDING.md#platform-list must be supported before this moves out of experimental (imo it's fine to land without initial support for some platforms if this feature is experimental and opt-in)

[jasnell]

To some of @bnb's questions:

what are the criteria for adding package managers to corepack
what are the criteria for removing package mangers to corepack
what are the criteria for evolutions of package managers (like Berry) being added to corepack
do all the current package managers meet all those criteria
are there any package managers that are left out despite meeting that criteria

I do not believe that these need to be answered before this lands and while it is still marked experimental. I would agree that these are exit criteria to graduate from experimental, however.

have we measured the impact on our consumers, like FaaS platforms or end-user developers
what is that impact on our users, beyond speculation

We really won't know until it's in user's hands, for that, we should land it as opt-in experimental.

can users turn this off

They should be able to, yes.

are we comfortable adding something this fundamental that has a single maintainer
are we as the Node.js willing and 100% able to commit, without question, to supporting this for the next decade

We haven't applied that strict of reasoning to other things and we shouldn't now. There's no guarantee that anything we add to Node.js is going to have people willing and able "without question" to support it for the next decade. That said, this is the whole reason for wanting to bring the project into Node.js and encourage additional maintainers. That said, this thing is really small and shouldn't need much effort to maintain.

if we land this, shouldn't it be behind a flag

Given that it's an install-time thing and not a Node.js runtime thing, the "flag" would be an option in the interactive installer to omit it, which absolutely should be the case.

[zkochan]

are we comfortable adding something this fundamental that has a single maintainer

I think there will be no lack of maintainers. This feature would be really useful for pnpm users, so I'll help out if there'll be a need. I believe other pnpm users/contributors will be willing to help as well.

[isaacs]

I have some concerns.

1. What's the story for self-updating? Ie, if someone runs npm install -g npm@latest, and "npm" is a shell script that calls pmm npm "$@", then self-updating will be broken without specifying --force. If that is not addressed, it presents a significant security concern, as we can no longer ship urgent patch upgrades to package managers (or even new features) without a new node release or requiring that users manually remove the pmm shell shim.

2. What's the story for updating the version that pmm has in its "known good" set? Surely, pmm's knowledge of npm/yarn/pnpm version goodness will be less than that of the maintainers of these projects who triage and investigate bugs on a daily basis. If self-updating is blocked (or even just changed from the currently documented flows), then this will become a huge hassle almost immediately.

3. How is this better than just shipping all three package managers, and providing an option in the msi/pkg installer to choose them as optional components? That seems much simpler, a less complicated user experience, an equivalently level playing field, and neatly addresses the two concerns above. If the objection is "smaller core", well, that ship has sailed. The optimal solution there would seem to be "no package manager", which no one wants. Shipping a new shell script shim which installs any of 3 package managers, and can't be easily updated, feels like we're making core reduced on bytes, but vastly bloated on complexity.

[jasnell]

What's the story for self-updating

Well, to be clear, corepack does not currently replace the npm binary. So there's no immediate risk there. That said, the corepack prepare and corepack hydrate commands cover updating the package manager versions. Newer versions can also be pinned in the package.json for a project.

What's the story for updating the version that pmm has in its "known good" set?

Pull request... No different really than updating the npm client version in core.

How is this better than just shipping all three package managers,

People who do not want yarn installed should not ever need to have it. Shipping all three would bloat the installer image needlessly, and providing an option to download on install is really no different. This gives the additional benefit of allowing different projects to select which package manager they require, which an install time only option would not provide.

and can't be easily updated

Not sure what you're basing this on. The design documentation for corepack goes into detail on how both corepack and the package manager would be updated and the flow is not much more complicated than what we have currently.

[ljharb]

@jasnell which yarn exiting zero is "having it".

[mmarchini]

I don't think anyone suggested how to make this opt in for v15 yet, and imo if it's opt out it's definitely a semver major change (especially with @ljharb point on which). My suggestion is to initially ship it with Node.js without creating the scripts for each package manager, and then users who want to opt in the behavior can run pmm init/pmm opt-in/node --experimental-pmm-opt-in (those are some suggestions but you get the gist).

[arcanis]

What's the story for self-updating? Ie, if someone runs npm install -g npm@latest, and "npm" is a shell script that calls pmm npm "$@", then self-updating will be broken without specifying --force

As @jasnell mentioned the npm binary is untouched by this PR, and I personally don't intend to change that, as I don't feel it's my place to make any critical change about a competing project (plus it comes with its own set of considerations unique to its situation). My only goal here is to ensure that my users' interests are accounted for - nothing more, nothing less 🙂

What's the story for updating the version that pmm has in its "known good" set?

The LKG releases bundled with Node will be updated by pull request (just like npm is updated by pull request). The LKG releases that apply on an individual user's machine (typically what people would change by running npm install -g) can be updated by this user using --activate.

Note that npm install -g will keep working, as it will overwrite the Corepack symlink with the regular one (meaning that it won't break the workflow for people running this command right after installing Node). @isaacs If you're open to collaboration please refer to this topic, as we can probably slightly improve the npm install -g DX together (right now it works because Yarn workarounds a small issue with npm).

My suggestion is to initially ship it with Node.js without creating the scripts for each package manager, and then users who want to opt in the behavior can run pmm init/pmm opt-in/node --experimental-pmm-opt-in (those are some suggestions but you get the gist).

(note: pmm is now Corepack, since pmm was already taken on the npm registry)

My main worry with making it opt-in is classic: if there's a problem, we will be less likely to know about it - meaning that once we reach the end of the experiment, even if nothing bad happened, we still won't have enough confidence that the workflow will be stable enough for an even release (especially since Node wouldn't monitor how many people opted-in, so we would have to assume that it would be a very low amount).

By contrast, seeing real-life feedback for an opt-out would guarantee that nothing sneaked by us, allowing us to safely make informed decision later down the road. And of course that wouldn't prevent us from having corepack disable, just in case.

[richardlau]

People who do not want yarn installed should not ever need to have it. Shipping all three would bloat the installer image needlessly, and providing an option to download on install is really no different. This gives the additional benefit of allowing different projects to select which package manager they require, which an install time only option would not provide.

On this point there is a difference between shipping the package managers in the installer image vs option to download on install in that the former allows installation on a machine without an internet connection.

[jasnell]

@arcanis, based on the feedback, the concrete changes I'd like to see before moving this forward:

1. Initially, let's land it as opt-in... that is, require a flag for installer.py to enable it. This will give folks an opportunity to experiment hands on and identify any potential issues. The idea would be to switch that to opt-out after a couple of minor release cycles once it has been demonstrated that folks fears of potential issues have been relieved.

2. Let's implement the command to remove/disable a particular jumper. Something like a corepack disable ${name} should do the trick. In addition, let's make it possible to specify these in the installer.py command line arguments so that a particular jumper can be disabled immediately on installation. Similarly, a corepack enable ${name} should reenable a particular jumper.

Given that this does not impact npm at all, there really shouldn't be anything further to discuss there unless the npm team decides to adopt it.

[devsnek]

also corepack disable to disable everything

[mmarchini]

The idea would be to switch that to opt-out after a couple of minor release cycles once it has been demonstrated that folks fears of potential issues have been relieved

I still don't think we can switch from opt-in to opt-out in a semver minor (not as it is today at least). IMO this is not something we should rush as it has the potential of breaking many automated workflows via scenarios we didn't consider so far.

[joshmanders]

For give me if I am wrong, just trying to understand things, first I love the idea of a level playing ground for all package managers, but why not just have --with-npm or --with-yarn as flags on install and let the user decide at install what package manager they want as their default and then go from there?

If I use npm for everything on homebrew. Ican just do brew install node --with-npm and then now I have npm, or brew install node --with-yarn and now npm doesn't exist, but yarn does.

I am worried about the complexity and UX of having all these extra commands and things going around. I know myself using this would be trivial, but my co-worker who updates our gatsby site and only knows what commands we teach her and often messages me because npm start failed because she forgot to cd into the right directory would add TONS of extra confusion for her.

[devsnek]

That would more or less be something that brew does, not that we do. Our official distributions, aside from the windows installers, are just files that you copy, and you're free to not copy the npm-related files.

[zkochan]

This message might answer your question: #15244 (comment)

Which package manager to use is not up to the user but rather up to the project. All developers that work on a given project should use the same package manager.

[isaacs]

Ok, so if corepack doesn't install a npm "jump script" as the proposal suggests, then you're saying that someone installs node, and runs npm and it's not found?

I'm not sure how this needle is being threaded here. Either there's an executable in the PATH called npm, or there isn't.

[mmarchini]

Unfortunately approach doesn't fit any Linux installed that I'm aware of (and we don't maintain the brew recipe for Node.js either). It also doesn't fix the problem that "I want to use npm but this project I'm working on only works with yarn".

I am worried about the complexity and UX of having all these extra commands and things going around

The idea is that no extra commands will be required (you need to run yarn? You just run yarn, if it's not installed it will install it for you). There was a demonstration of this tool on the Aug 27 TSC meeting for anyone who wants to see how it works.

[devsnek]

npm is still packaged in its entirety, not part of corepack.

[mmarchini]

@isaacs there will be no changes to npm at this time. If a user runs npm, npm will run, that's it.

[isaacs]

Ok, so the idea is that node ships npm and corepack, with a yarn jump binary that installs yarn the first time you run it?

What happens when someone runs npm install yarn --global? (If that jump binary is at /usr/local/bin/yarn, and it is not a symlink to the bin in ../node_modules/yarn it'll refuse to clobber it. I believe yarn has the same protection in place.)

If that's the case, then npm can self-update and yarn can't (and can't be installed by npm either). Not exactly a level playing field.

[arcanis]

Note that npm install -g will keep working, as it will overwrite the Corepack symlink with the regular one (meaning that it won't break the workflow for people running this command right after installing Node). @isaacs If you're open to collaboration please refer to this topic, as we can probably slightly improve the npm install -g DX together (right now it works because Yarn workarounds a small issue with npm).

[isaacs]

@jasnell

and can't be easily updated

Not sure what you're basing this on. The design documentation for corepack goes into detail on how both corepack and the package manager would be updated and the flow is not much more complicated than what we have currently.

I'm basing it on the fact that npm i -g npm would not update itself, because it'd refuse to clobber an executable outside of its control. If npm isn't included in corepack, and node continues to ship the real npm, then that's not an issue for npm. But it is still an issue for yarn and pnpm, and all of the extant documentation on how to get yarn and pnpm (ie, by installing it with npm i -g.)

How is this better than just shipping all three package managers,

People who do not want yarn installed should not ever need to have it.

If you are shipping a binary named yarn that automatically installs yarn from the registry the first time it's run, then they have it. You've just traded bytes in the installer with bytes over the wire later.

Shipping all three would bloat the installer image needlessly, and providing an option to download on install is really no different.

It would increase bytes, but reduce complexity. I believe the tradeoff is better. Download time is less valuable than debugging time.

This gives the additional benefit of allowing different projects to select which package manager they require, which an install time only option would not provide.

Should this just be a thing npm can do? It wouldn't be hard at all to add an npm command that will run a command with whichever package manager is declared in the project, and/or raise warnings/errors if you've specified a package manager other than npm, with instructions on how to install the package manager specified.

[jasnell]

But it is still an issue for yarn and pnpm, and all of the extant documentation on how to get yarn and pnpm (ie, by installing it with npm i -g.)

The intent here is to change how both yarn and pnpm are installed and updated so this point is a bit moot. That is, npm i -g would no longer be the way either package manager is installed or updated.

[mhdawson]

I think we are getting pretty close to the summit so I'd ask that we hold off landing this until the summit or alternatively if waiting for that does not make sense, add to the TSC agenda for the next meeting for a discussion there.

[bnb]

@bnb are you happy that your questions that you raised have been addressed through the references above?

@mhdawson I mean, five of them were entirely dismissed as irrelevant for inclusion if we're shipping this as an experimental feature. I disagree with that dismissal because IMO this is a situation where once we ship this we'll never be able to back off of it and should therefore have the context defined before shipping, but I don't think I'll be able to meaningfully convince anyone of that.

[mhdawson]

@bnb I share the concern about once in as Experimental it will be difficult to back out. I don't think we should be dismissing questions on that basis. The answer to my earlier questions were that issues had been addressed and I was not entirely sure of that and it sounds like you are confirming that it is not the case. I'm hoping that in the discussion at the summit we can surface remaining concerns and get broader consensus on how to move forward.

[mmarchini]

+1 on waiting for the summit (marking explicitly as request changes so we don't land by accident)

[pedrofurtado]

Any news about it? 👍

[mhdawson]

In terms of "The corepack binary is included by default but not yarn / pnpm, so users have to opt-in via corepack enable"

To confirm this this means that unless I opt-in via corepack enable, running yarn will result in a "not found". I'm very uncomfortable with "silently downloading" binaries without an opt-in by the user.

[aduh95]

@arcanis This needs a rebase to resolve the git conflict.
@mmarchini Is it OK to dismiss your change request now that nodejs/TSC#1012 has been voted?

[arcanis]

@aduh95 I was thinking about reopening a fresh PR with the recent updates in the Corepack repo, along with a few other I plan to do (like the rebase you mention), is that ok? Discussions in this thread have mostly be meta, I think it would benefit from a reset focusing on its implementation.

[trivikr]

I was thinking about reopening a fresh PR with the recent updates in the Corepack repo

@arcanis Any update on the new pull request for corepack?
I landed on this PR after reading your yarn 3 announcement blog post.

[arcanis]

Done - I've opened #39608 which updated Corepack and rebased it against the trunk. Archiving this PR! 🙂