Package management rfc

[jneem]

Rendered

[yannham]

First, thanks a lot for putting that together. I think it's clear and well-articulated. Something that might be improved a bit (I left a related comment) is to clearly state the goal and non-goal of what we're trying to achieve here, to delimit the scope more explicitly.

[jneem]

I've updated the RFC based on feedback here and discussion at the weekly.

[GuillaumeDesforges]

Why is it needed?
Should this be a version constraint? e.g. >=1, ^1, >1.1.3,<2 etc?

[jneem]

Ah, right. I guess version should be SemverConstraint (or VersionConstraint). nickel-version is the minimum nickel interpreter version required by the package. Maybe min-nickel-version is clearer?

[GuillaumeDesforges]

Funnily enough, Nickel Package Manager would be... npm ☠️

[GuillaumeDesforges]

This seems to be a huge argument in favor of embeding package management in nickel's CLI.

At some point, I can hardly imagine anybody not using package management. That would mean everybody would use plate eval instead of nickel eval anyways.

[aspiwack]

My initial comments (I'm coming at this completely fresh, I haven't read the discussion above)

[aspiwack]

It should be said that it's only “their” problem until the package manifest ends up in the package registry. Then, a manifest that takes time to compute may become everybody's problem if we're not careful. And will certainly become any of their dependents' problem.

[jneem]

Oh, this is one important thing I left out. The index entry in the registry (which is in json, not nickel) contains the dependency information from the manifest. So resolving dependencies for registry packages doesn't require evaluating their manifests. The slow evaluation only matters when they create their PR for uploading their package to the registry.

[aspiwack]

This certainly asks the question of what the semantics would bee. Presumably you couldn't import foo anymore, instead you'd import foo/"bar" or something.

[aspiwack]

This doesn't sound particularly meaningful, anyway. If we have a different tool to manage packages (e.g. apt), then we don't want electroplate packages to break. Instead, we want apt to pass the same packages that plate usually does.

[yannham]

I think the idea isn't that we switch package manager entirely for a fixed set of pavkages, but rather that some packages are managed by package manager A and we want to add other packages are managed by package manager B, typically provided as a bare flake output. More of a multi package providers scenario. At least when we looked at not having a proper package manger just for Nickel but relying on existing ones. Maybe it's less relevant now.

[aspiwack]

But you might have to switch package manager anyway. At any rate, I don't think the tool used to provide package is relevant for namespacing.

[aspiwack]

A missing alternative which I find tantalising, building on Nickel's expressiveness, is to not have an entry point file at all. Instead, the manifest has a field entrypoint which contains a Nickel value. And that value is what the package exports. You can always have entrypoint = import "main.ncl" to restore the shape that you propose.

In fact, we could even have

  entrypoint | default = import "main.ncl"

(as long as imports aren't evaluated at all unless forced, because main.ncl may not exist)

As you point out, it's isomorphic to your solution. But it does have one fewer baked-in name.

[jneem]

as long as imports aren't evaluated at all unless forced, because main.ncl may not exist

This might be an issue, because currently we do eagerly parse and typecheck imports.

[aspiwack]

This really doesn't seem like a very good argument at all. You can always release a version of Nickel which only updates the package manager interface. It's not like Nickel is very ossified and your rolling up a new tool. The bloat thing sounds even less convincing somehow.

An argument in favour of a separate cli may be that it guarantees that the Nickel cli can support integrating with external tools. The risk of baking the package management in is that it makes it possible to have the built-in package manager use capabilities of Nickel which aren't available to other tools, and it'd be bad for Bazel and stuff (but, of course, a just as robust way to ensure this is to maintain rules_nickel carefully).

---

I'm honestly not convinced about this separate package manager. Here's my reasoning: it works fine for Node, Rust, Ocaml, … because the vast majority of project who use Node, Rust, Ocaml, … are Node, Rust, Ocaml,… projects. I expect that actual Nickel projects will, in contrast, be a tiny minority: Nickel is to usually be used in a something-else project. Therefore it doesn't really make a ton of sense to pull in an additional project-management tool besides Npm, Cargo, Opam,… to manage just the Nickel bits.

On the other hand, we must make sure that we can call Nickel without the package management. But maybe we can flip the intuitive behaviour around nickel eval could be what you call plate eval while nickel simple eval could be the current nickel eval (since it'll usually be called by other tools). Or something. I don't know the design space is huge.

[jneem]

Ok, I'm redoing that section with the built-in package management as the proposed solution, and a separate command as an alternative.

[aspiwack]

An alternative is to never do any dependency resolution at all: target a particular commit of the registry, and just use the latest version of each package according to said commit. This is the Nixpkgs way (and also Stackage). It has pros and cons, but it has a fairly clear story for lockfile updates (i.e. just create lockfile entries for the new packages, never touch the existing package (unless asked to by the user, of course)).

[yannham]

The lack of dependency resolution in flakes leads people to implement their own layer (https://discourse.nixos.org/t/introducing-flakehub/32044) which the usual lot of community split, bikeshedding and drama. I don't know about the Stack situation, though.

[aspiwack]

I purposely didn't speak about flakes. Maybe inviting this comment 🙂 . I would argue that the lack of name resolution is a feature of flakes: you only depend on a couple of flakes, the rest, you get from Nixpkgs, this does means that anything not in Nixpkgs is a second class citizen (and I very much believe it's the right thing for Nix). Same with Stackage: you depend on a version of Stackage and possibly a couple other packages (including packages which are in Stackage, but you want another version), whose version you precisely specify.

I don't know which of resolution or no resolution is best for Nickel. But I'd like to hear some arguments.

One of my reason to like no-resolution is that bound management is a lot of busy work. And bounds are always wrong. But one of the thing that makes Nixpkgs and Stackage work, is that it's possible to detect when things don't build together, notify their maintainers, and fix it. Maybe it's harder with Nickel, I don't know.

[yannham]

I think that the flakes and Stack models rely on the existence of a globally coherent versioned set of packages (is that what Stackage is)? In Nixpkgs those are NixOS releases. I agree this is usually great from a user experience point of view, but also a non negligible burden put on the people in charge of curating, updating and releasing this set.

[aspiwack]

Yes, Stackage is a collection of so-called snapshots. Each snapshot has at most one version of each package, which are tested to be able to build together.

I think one of the biggest obstacle that you'd have for Nickel is probably the testing bit. Since you don't necessarily have types to guarantee that everything works together well (and evaluating everything like Nixpkgs does is pretty prohibitive).

Possibly the approach of allowing several copies of the same package alleviates a large chunk of the issues with bounds and resolution. But I think this alternative should at least be addressed in the document.

PS: if we go with version bounds, we need to have an answer to the question: what is the process when we discover after the fact that a bound is too permissive?

[jneem]

I added an "Alternative" section about global snapshots as a placeholder, but it's probably worth discussing it in more detail in the weekly.

Anecdotally, I find that the rust defaults (dependencies use the "^" mode by default, and you can have multiple copies of a package that don't share the same semver bin) work pretty well. The point is that the default dependency bounds align with the no-duplication bounds, and so if those bounds are too restrictive then you just end up with multiple copies of a package.

The bounds are usually not too permissive, because they are then someone has done semver wrong. But if it does happen, you can often work around it by pinning a dependency. For example, suppose I have a dependency on dep, which in turn depends on dep2 ^0.1.0, but then dep2 releases a broken 0.1.1. I can force version resolution to fall back to the old version of dep2 by adding a dependency on dep2 =0.1.0.

[aspiwack]

I'm sorry, my question was ambiguous. Here's the scenario that I have in mind: we have packages A<-B uploaded in the registry. And we discover that the latest version of B, says it's compatible with versions of A which it actually isn't compatible with (whatever the reason). We now need to update the registry somehow. It could be by altering the how the current version of B depends on A. Or it can be by marking this version of B as incorrect somehow, so that it's never picked up by the solver (and uploading a new version).

But, one way or another, we need to make sure that the bad combination is forbidden, otherwise, it'll be possible in the future for the solver to pick that combination, even if there are newer versions of B.

We need policy and workflows for to handle this case.

[jneem]

Ah, right. I'll add a section on that. I like the "mark B as incorrect" option, because it can also be used to work around other kinds of breakage.

[aspiwack]

Or github.com/nickel-lang/nickel-mine.

[aspiwack]

This representation strategy means that the manifest files (hence the packages' metadata) isn't kept in the registry. It means that querying the registry can be quite expensive since it requires many calls to many Git repositories. Is this reasonable? (genuine question, I don't really have an opinion on this)

[jneem]

This is also related to the thing I left out, that the registry index has enough information for dependency resolution. We don't store the whole manifest in the registry, but enough that we can do version resolution just by querying the index (and manifests from git/path dependencies, of course).

[aspiwack]

But maybe we don't just want dependency resolution? What if we want to query package descriptions, for instance? What meta-data should be in the repo is something we definitely ought to answer, oughtn't we?

PS: I'm not even sure “oughtn't we” is the right way to end this sentence. But it sounds super cool, so I went with it anyway!

[jneem]

Fair enough. I guess the index ought to include whatever metadata we want to be easily queryable.

[aspiwack]

Opam manages very well by pull requests. The cli can automate writing a pull request for you. If the volume becomes too high, we can also automate merging the pull request. This sounds way easier than having to maintain a backend service. So I'm heavily biased toward this alternative.

[yannham]

Yes, I think this was the general feeling when discussed orally in the weekly: this is the simplest and most reasonable way to start.

[aspiwack]

There are two big technical missing bits in your RFC:

• How does the nickel command line understand packages? This is your package-map bit, but extended beyond the interpreter: how do we provide said package map to the interpreter to begin with?
• Where/how are the packages stored on disk (I think this is easier than for most programming languages since we store the packages verbatim, so we don't have this thing where the same version of the same package is actually two different artefact because they were built with two different compilers).

[jneem]

• I've added some more details to the section on the CLI interface
• I added an extra section at the end on how we store packages

[github-actions]

Bencher Report

Branch	1983/merge
Testbed	ubuntu-latest

⚠️ WARNING: The following Measure does not have a Threshold. Without a Threshold, no Alerts will ever be generated!

• Latency

Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the --ci-only-thresholds CLI flag.

Click to view all benchmark results

Benchmark	Latency	nanoseconds (ns)
fibonacci 10	📈 view plot ⚠️ NO THRESHOLD	503,920.00
pidigits 100	📈 view plot ⚠️ NO THRESHOLD	3,229,400.00
product 30	📈 view plot ⚠️ NO THRESHOLD	809,160.00
scalar 10	📈 view plot ⚠️ NO THRESHOLD	1,495,200.00
sum 30	📈 view plot ⚠️ NO THRESHOLD	799,940.00

🐰 View full continuous benchmarking report in Bencher

[aspiwack]

Thanks for the updates. A few extra comments below.

[aspiwack]

It probably is, to be honest. But on the other hand, you'll have to be careful to avoid making “weird”. The shortcut and the full method should look decently similar. Or something.

[aspiwack]

Storing a content hash is a good idea. But there's more than one way to do so, so you may end up not getting much more compatibility than you planned.

That being said, there is one content-hash standard, namely SWHID from our good friends at Software Heritage. It's specified here https://www.swhid.org/specification/v1.1/ . Maybe worth looking into it?

[jneem]

It looks like SWHID is very easy to support. As far as file and tree identifiers go (which I think are the ones we care about; the registry only needs to identify the tree, not the whole history), it's 100% git-compatible. That is, if the tree if is d198bc9d7a6bcf6db04f476d29314f157507d505 then the SWHID is
swh:1:dir:d198bc9d7a6bcf6db04f476d29314f157507d505

[aspiwack]

Do we need some garbage-collection/cache-eviction policy?

[jneem]

Maybe, but it might also be ok to punt on it. The local cache doesn't have an index, so it's safe to just delete things manually.

[aspiwack]

I'm sorry, my question was ambiguous. Here's the scenario that I have in mind: we have packages A<-B uploaded in the registry. And we discover that the latest version of B, says it's compatible with versions of A which it actually isn't compatible with (whatever the reason). We now need to update the registry somehow. It could be by altering the how the current version of B depends on A. Or it can be by marking this version of B as incorrect somehow, so that it's never picked up by the solver (and uploading a new version).

But, one way or another, we need to make sure that the bad combination is forbidden, otherwise, it'll be possible in the future for the solver to pick that combination, even if there are newer versions of B.

We need policy and workflows for to handle this case.

[aherrmann]

Very interesting proposal!
I've added some comments from the Bazel perspective.

[aherrmann]

I didn't see it mentioned explicitly, so I wanted to raise it. Will import be forbidden in the manifest itself? I would imagine that that would be a good idea. But I haven't thought about it very deeply.

[jneem]

Good question, I had originally thought that the manifest wouldn't support package management, but it would be allowed to import normal paths. But maybe let's start by forbidding imports altogether.

[aherrmann]

Such features would be great for integration with other package managers or build systems.

[aherrmann]

Are you considering support for custom registries?

When Bazel introduced its new dependency manager bzlmod they also allowed users to define custom registries and even use multiple registries. Commercial users often like this because they can place proprietary code into their own private registry.

[jneem]

Good point, I'll look into that.

[aherrmann]

Just for reference, the Bazel bzlmod solution is to use Go's MVS and fail if they encounter incompatible version requests for the same package. But, the root package can declare single or multi version overrides.

[jneem]

That's a very interesting read, thanks! This approach and the one of allowing duplicate semver-compatible versions seem to be pretty much mutually exclusive, so I guess we need to pick one or the other?

[aherrmann]

I guess the user controlled override is compatible with either approach. But, otherwise, yes, I think so.

[aherrmann]

Build systems like Bazel prefer to do the fetching themselves. Is the intention that plate can just produce the URLs for dependencies but leave the fetching to an external tool?

[jneem]

Yes, I think we can have a nickel package list-all-urls (or something) subcommand that outputs the information needed to fetch dependencies. Would that meet the needs of build systems?

[yannham]

Reading again this section, I wonder if version resolution and snapshots are necessarily incompatible. We could get started with version resolution but at some point provide a snapshot to take stuff from, I suppose. A bit like foo.workspace = true in cargo, but with snapshot-foo instead of workspace.

[yannham]

I wonder how useful this is. That is, I don't expect it would happen a lot that there's a unique version that can be picked and it's a yanked version. Often either the yanked version is immediately followed by a backward-compatible update - minor or patch -, or the yanked version wasn't a X.0.0 version and there are other previous versions of the same major version.

I think a yanked version would be the only choice if people started to depend on this specific version too fast (otherwise a previous version of the same bin could still be picked) AND the correction is a breaking change (otherwise the more up-to-date correction could still be picked). Which doesn't sound common enough to mandate an ad-hoc solution, IMHO.

[yannham]

I think I don't understand: you say it's a global setting, but we still configure it at the level of the manifest of a package? Or by global do you mean it applies to all the dependencies of one package, instead of potentially one of the dependency of one package?

[yannham]

Should we have any form of security here? Adding a source-replacement has the potential to substitute an existing package with pretty much anything. The security model is not as bad in Nickel as in say Nix, which can install arbitrary executable or inject hidden malware in your glibc, but in Nix you do need to add a public key and a there is mechanism to make sure the user really intended to use a substituter (need to be a trusted user, and will ask on the CLI if you want to use the substituters configured at the flake level).

The answer might be that we don't care in our case, I'm thinking out loud here.

[jneem]

I think it isn't a security risk because source replacement is only configurable at the top-level: it can't be triggered by some package deep in your dependency tree. If the top-level package wants to be malicious, it probably has sneakier ways of being so.

[yannham]

I wonder what's the cost of doing that on each evaluation. As opposed to compiled languages, we might have to pay the dependency resolution step on each run, including for small-ish configurations (as opposed to say cargo when you pay the price at build time but not at run time). Do you think that would be an argument in favor of better separated package-management, where nickel eval would just pick the existing lockfile or fail but not perform the update automatically? Or at least an argument for not updating the lockfile automatically? Or do you expect those costs to be negligible?

[jneem]

Good question, I guess we'd have to measure it to be sure. I expect version resolution to be quick, especially if there's an up-to-date lockfile but I don't actually have data.

Having to manually trigger a re-resolution is definitely an option; I think poetry and npm do this. I personally find it a bit annoying, as it's an extra step that's easy to forget.

[yannham]

After the last review, and discussion in team meeting, we decided that the RFC is in a good state and answered most direction questions. We shall go forward with this version (although @jneem has minor updates to do still before merging).