Support alternative version pinning mechanism

[dcramer]

NOTE: this issue predates this project's rename to Volta.

We love the idea of Notion (and nvm) as it makes it easier to persist a shared version of Node.js throughout many environments (local, Travis, production [Docker]). However, there's a limitation in Notion that we didn't have with nvm, and that comes from locking node/yarn within package.json itself.

This limitation realistically only comes up when you're doing layer caching, which if you're not familiar with I'm probably not a good person to learn from, but here we go:

We try to install Node and capture it outside of package.json because it means we dont have to rebuild the Node.js layer every time packages change. Dockerfile looks something like:

COPY .nvmrc /usr/src/zeus/
ENV YARN_VERSION 1.7.0
RUN set -x \
  && export NODE_VERSION=$(cat /usr/src/zeus/.nvmrc) \
  [...]

Now this worked actually extremely well. We had a single file we could pull/parse to get the node version everywhere.

When moving to Notion our only real alternative is passing a Docker build arg:

--build-arg NODE_VERSION=$(shell bin/get-node-version | tr -d '\n') \

This is fine and all, but tools like Google's Cloud Build (cloudbuild.yaml) doesn't let you do dynamic arguments like this. That means we're stuck with either rebuilding the Node.js layer every time package.json changes, or hardcoding NODE_VERSION elsewhere.

I'm not proposing a solution here, but wanted to make sure everyones aware of this problem as its a shortcoming with a lot of package management tooling.

[dcramer]

/cc @mitsuhiko who also has been playing with Notion on some of our other projects and might have additional feedback

[charlespierce]

Hi @dcramer, thanks for bringing this issue up! In our design discussions, we have always wanted to avoid adding a notion configuration file to a project, as we didn't want to add yet another required file that everyone who uses the tool would have to check in to source control.

It seems like, for this use case, you would prefer a separate Notion configuration file, so that you can use that as a single source of truth that is distinct from package.json? If that's the case, it seems like a reasonable compromise would be to allow Notion to handle both cases: Configuration inside of package.json as well as configuration in a separate config file. We'll need to hash out the details of precedence and how to handle the notion pin command, but that could provide a possible solution.

Another possible option, if I'm interpreting your concern correctly, would be to have a Docker image that has Notion installed, and then let that Notion install handle fetching node the first time it is activated. Would that work for your situation as an alternative?

[dcramer]

@charlespierce the docker image wont help us (we need base images and then to layer things on top). The main concern is that we want to be able to define a version via a filesystem object that is not in package.json. The reason is that we can hash(that file) to know if Node.js needs updated or not, whereas doing that package.json will suggest Node.js needs updated almost every day based on unrelated dependency changes.

In general I think this is going to be an ongoing industry-wide concern, but I've seen it's often not been approached in a way that works well with existing systems.

The alternative is that whenever we change our Node.js or Yarn versions we have to update multiple locations. This issue is on the goal of trying to accomplish a singular canonical version reference, without compromising build caches (and thus build time/resources).

If Notion decides "package.json is the standard and we dont want to have multiple locations" I think that's totally fair. Again, to me this is a general industry-wide problem that very few package management solutions have resolved. We got someone lucky that nvm was doable in a manner that worked for us, but Notion is easier for developers to work with out of the box.

[chriskrycho]

@dcramer you're not alone in that desire; I know of at least one other team for which that would make a big difference (my old team, which is also making good use of Docker in a not-dissimilar way)!

[chriskrycho]

Related: volta-cli/rfcs#33.

[charlespierce]

FYI With the change to support extends in package.json (#755), this will be sort of supported. While intended for workspaces / monorepos, Volta's logic for extends is flexible, so you could use it like this:

package.json:

{
  ...
  "volta": {
    "extends": "./volta.json"
  }
}

volta.json

{
  "volta": {
    "node": "14.0.0",
    "yarn": "1.22.0"
  }
}

So that changing your Node / npm / Yarn version wouldn't require making a change to package.json, it would keep the same contents. If you also happened to be in a monorepo, you could then set extends in volta.json to point at the workspace root, and that would work as expected.

The main caveat with this is that with how volta pin works, you wouldn't be able to use that as a convenience method and would need to manually update the version in your settings file (volta.json in this example). Running volta pin would result in the version being written into package.json along side the extends directive.

[frangio]

I'm personally interested in this feature so that I can use Volta in my personal local environment without polluting the package.json of my open source projects. So the extends option doesn't work for me.

[charlespierce]

@frangio Understandable! While the approach I outlined is a solution to this issue, I don't think it's the solution since there are a number of problems with the usability.

Can you expand on your use case a little? Are you looking to have a completely separate file that you can add to .gitignore so that it isn't committed to the open source projects? Or are you looking to have that file completely outside of the project folder?

The former is potentially doable, though we'd need to make sure we're on top of performance and not trying to look in too many different places for files. The latter is difficult and I'm not sure really fits into the Volta data model.

Out of curiosity, what is the hesitation regarding adding volta to the package.json for the open source projects?

[frangio]

Are you looking to have a completely separate file that you can add to .gitignore so that it isn't committed to the open source projects?

Yes exactly. The performance concern is very interesting though. Maybe there could be a limitation that this separate file has to be next to a package.json file. Sounds kind of arbitrary but it would limit the effect on performance.

The hesitation is because of a sort of "separation of concerns". I view Volta as a concern of my local setup, and as much as I love it and would recommend it to everyone, other developers may choose nvm (or anything else) and it wouldn't make sense to have duplicate configuration for all of these tools. I would've thought they'd all use the engines field, but I saw there are reasons for Volta not to use that (which I haven't read).

[charlespierce]

Related proposal from Twitter: Support the .nvmrc file directly so that users of Volta and NVM can both work on the same code without changes. There's a few issues we'll need to tackle / design in order to make that work:

• Where to look for the .nvmrc? My preference here would be as a sibling of package.json iff that file exists and there isn't a volta key already. We want to minimize the file I/O operations on the hot path, to make sure that our startup time is quick.
• How to map the values supported by .nvmrc into Volta's VersionSpec type? It appears that nvm supports a number of different keywords that don't necessarily match up with Volta's.
• How to handle range versions? .nvmrc supports a range / semver specifier (e.g. 4.*), how should Volta treat that? This is the biggest hurdle that I can see, because Volta has a fundamentally different model from nvm for how to manage the local cache.
  • We don't want to fetch the Node index and re-resolve the version every time a Node command is run, since that will add unnecessary overhead to every command.
  • At the same time, if we only check the local cache, we go against Volta's philosophy of auto-downloading the necessary versions on-demand and require the user to manually manage their "installed" versions.
  • Even a hybrid model requires the user to be aware of the internal cache, which they shouldn't need to manage directly.
  • Perhaps we only support pure versions in that file, and show a warning if a file would be used but is being skipped because we don't have a specific version.

If we can work out the specifics of how to handle the .nvmrc file, the existing extends logic already supports multiple sources of truth, so it should be possible to slot .nvmrc into that logic. Similarly, the existing extends work also should support a new, separate file (perhaps .voltarc.json?) that can be loaded.

[ljharb]

Hi - nvm maintainer here.

The way nvm looks for an .nvmrc file is just like npm looks for .npmrc or eslint looks for .eslintrc - in the current directory, or in any ancestor, all the way up to /. Any other algorithm would be incorrect.

.nvmrc contains a "version-ish". This is not actually a range - it always resolves to a single version, either locally, or remotely, depending on the context/command. Asterisks are not supported whatsoever. For example, 4 means "the latest version of 4", or in semver terms, "^4.0.0". "node" means "the latest version of node", or in semver terms, "*".

I beg you not to support .nvmrc at all unless you can support 100% of its semantics, which include alias lookup, release line lookup, LTS lookup (lts/*, lts/argon, etc), major (4), minor (4.2), or exact (4.2.1).

[charlespierce]

Hi @ljharb, thanks for the clarification on the specifics of .nvmrc! I agree we shouldn't try to support .nvmrc unless we follow the same semantics, as that's just going to create undue confusion all around. And given that, ultimately I think the non-exact version-ish resolution is going to make supporting .nvmrc untenable with Volta's current model.

The difference as I understand it (and please correct me if I'm wrong) is that nvm has an explicit command (nvm use) to say "Resolve the version now and set that up" after which it stays fixed (at least until the command is run again). By contrast, Volta is always re-evaluating the context, so we would need to constantly redo that resolution. Which would ultimately cause more problem than it solves, imo.

[ljharb]

The difference as I understand it

That's correct, yes - nvm use always uses the locally installed version list, and thus, shouldn't change unless someone has ran nvm install or nvm uninstall, or changed an alias. nvm install also reads from .nvmrc to install the desired version, and reads from the remote list.

[pho3nixf1re]

There is also the .node-version file which is supposed to be more generic between projects but the semantics are the same as nvmrc I believe. As an alternative, could Volta provide a command to migrate an nvmrc file over to the volta config automatically? This would be helpful in the case of migrating lots of projects at once or even using volta beside nvm for larger teams.

[bmblb]

Pinning to package.json pollutes the repository with irrelevant info. I'd love to have smth like .voltarc config somewhere in the directory to pin versions there. I wouldn't expect it to be compliant with any other tool, like nvm, just own config. @charlespierce would you consider that?

[ljharb]

@pho3nixf1re the semantics aren’t the same, for one because there are no agreed-upon semantics, but also because nvmrc supports anything nvm considers a “version-ish”, which includes user aliases, “iojs”, LTS aliases, etc.

[pho3nixf1re]

Sorry, let me clarify. I mean "the same" as in .nvmrc == .node-version. As for Volta my suggestion was to provide a command that would resolve the nvmrc once and then run the pin command based on that. It would be a one time thing to be used as a migration to Volta or to provide a temporary path for a team that has already established nvm.

[alexilyaev]

A separate config file support would be most appreciated.
Some projects already are using .nvmrc or .node-version and don't want to add another configuration to package.json.
Ideally in these projects I could add a .voltarc file (git ignored) next to the package.json or in a parent folder and that would be used in the absence of volta settings in package.json.

The use cases are also mentioned in #924.

I understand the performance concern, so maybe it can be an opt-in feature.
If it's opt-in, it could be even faster if we don't need to parse package.json first.

Side note:
I found Volta while reading The State of JS 2020 😄
https://2020.stateofjs.com/en-US/technologies/build-tools/

[matthewadams]

Volta is well on its way to be exactly what I'm looking for. The one feature that I'm missing is the .voltarc, searched for from the current directory on up to the root.

I really like the features offered by asdf, particularly https://asdf-vm.com/#/core-manage-versions?id=set-current-version, where you can set the version at many levels: a global default version, a local version (directory-based .voltarc, as has been discussed here), a shell version (via an environment variable like VOLTA_${TOOL}_VERSION), or even a version for an individual command ($ VOLTA_NODE_VERSION=12.15.0 npm run foobar).

If volta could support these same features, it'd be just jim-dandy.

[shadowspawn]

For possible interest, collection of .node-version usage: https://github.com/shadowspawn/node-version-usage

[jakub-g]

Additional data point: in our project we do have .node-version as canonical reference, and .nvmrc which symlinks to it (so only one file needs updating when bumping version of node).

The situation with volta is a bit different though because it allows pinning more than node but also the package manager.

[stefanhoth]

The situation with volta is a bit different though because it allows pinning more than node but also the package manager.

Wondering in how far it's still relevant to manage the package manager through Volta given that corepack exists now.

Additionally, Yarn commits it's binaries to the repo now as well, pinning them effectively.

Is there any benefit of keeping this extra functionality that is reducing the compatibility with nvmrc?

[iki]

Love Volta, on Windows it's probably the only sane Node version manager allowing me to work with multiple projects/components smoothly. That said, I don't want (and sometimes just can't) push that settings to all projects, so this is a real blocker, as both untracking package.json or keeping its local changes when switching many branches is not something that one can work with fluently.

+1 for .voltarc / .nvmrc / .nodeversion / whatever works best for you and doesn't have to be committed to repo, and no need to search upper folders, it's just enough to test if it exists in the same folder as package.json

[christiannaths]

Sorry to pile on, as a newcomer to Volta I just want to add my 2¢.

IMHO I think it's become expected behaviour to have tool-specific configuration files in the root of a repository (monorepo or otherwise). These feel like fairly well-established patterns at this point (see: eslint, prettier, nvm, etc.)

The majority of these tools all follow similar conventions:

• config is in json format
• a configuration file named so that it's unmistakable what tool it's for
• a package.json key as an alternative method of configuration
• if a config is found at the cwd, it is used. If not it searches up the tree until it finds one, ending at the user's home directory

So, given this pervasive pattern already exists, I guess I expected that it would be in place here as well. If I were a newcomer, I would not expect for Volta to read from an .nvmrc file, as nvm is a separate tool entirely. If there were a more generalized convention (looking at you .node-version) that became more established, then I'd also be delighted to see support for that as well.

I feel that package.json-only-config doesn't work exceptionally well for a tool like this, as it must be committed to the repository and that can violate team standards. Without an alternate config method I'm forced to reach for something else 😕

By the way, thank you for this amazing tool I feel like it's the answer I've been looking for for quite some time! 👏👏👏👏👏

[0xdevalias]

If I were a newcomer, I would not expect for Volta to read from an .nvmrc file, as nvm is a separate tool entirely. If there were a more generalized convention (looking at you .node-version) that became more established, then I'd also be delighted to see support for that as well.

Related to this, here was my /2c as shared on a different issue:

Another alternative would be to use the .node-version file format that many other tools have used for quite a while, and continue deprecating the volta key in package.json.

Or if you wanted to use a slightly less common, but more versatile standard name, asdf's .tool-versions would seem a good choice.

• https://asdf-vm.com/

  • Manage multiple runtime versions with a single CLI tool

• https://github.com/asdf-vm/asdf

Originally posted by @0xdevalias in #987 (comment)

[0xdevalias]

Another alternative would be to use the .node-version file format that many other tools have used for quite a while, and continue deprecating the volta key in package.json.

Or if you wanted to use a slightly less common, but more versatile standard name, asdf's .tool-versions would seem a good choice.

Related to the above, I just stumbled upon this repo that supports loading the version from a number of locations, in a specified hierachy. It might be overkill for the needs here, but I figured I would mention it in case it's useful:

• https://github.com/ehmicky/preferred-node-version

  • Get the preferred Node.js version of a user or project.

  • This looks for (from highest to lowest priority):

    • Any .n-node-version, .naverc, .node-version, .nodeenvrc .nvmrc or package.json (engines.node field) in the current directory, parent directories, or home directory

    • Any NODE_VERSION, NODIST_NODE_VERSION environment variable

Originally posted by @0xdevalias in #987 (comment)

[FarazPatankar]

For anyone else who ends up here looking for a possible workaround while this is implemented/resolved, I already had an alias set up to make it easy to navigate into my repo using ZSH and I just changed it to:

alias mono=cd ~/PATH_TO_YOUR_REPO && volta install node@14.18.1 yarn@1.19.0

Now, whenever I hit mono to navigate into the repo, my preferred versions are selected by default.

[johnhunter]

Further to @christiannaths comment I like the idea of a root .voltarc file

I'm a big fan of Volta and have found pinning really useful where I can convince the whole team to adopt it. But since starting to contribute to a large well established OSS project I've realised the lack of pinning inheritance from the root of a monorepo is an issue.

To have pinning work consistently across this project I will have to add volta extends to about 100 package.json files. Since many contributors do not use Volta I'm not sure that is going to be an acceptable change.

[collinstevens]

A .voltarc file would greatly ease the burden of working on projects which refuse to use allow a volta entry in their package.json.

I would be able to add the file to my .git/info/exclude file and never have to worry about different tool versions again, regardless of a projects inability to add a configuration.

[strokirk]

Would a PR be appreciated for adding support of .voltarc? Or is more definition and discussion necessary?