introduce declaring zig version ranges in packages; zig package manager additionally perform zig compiler version management

[andrewrk]

With the upcoming package manager (#943) there is planned to be a declarative file format, such as zig-package.json, which sits alongside build.zig and lists information such as the set of packages that are possibly depended on.

I propose that there are additionally these two fields in that declarative file. Example:

    "zig_version_min": "0.8.1",
    "zig_version_max": "0.9.0-dev.1703+6afcaf4a0",

Both fields would be optional. If min is omitted, it means there is no zig version that is too old. If a max is omitted, it means there is no zig version that is too new. The intent is that a maintainer would add a min if they discover that their package depends on an issue that has been resolved only after a particular zig version. They would add a max if they discover that a version of zig has done something backwards-incompatible that breaks their package.

When using the zig package manager (via the zig pkg subcommand), it would detect these fields, and, if found, acquire a suitable Zig compiler version.

Acquiring a zig compiler version would be done with different strategies:

1. (first priority) If the zig version used to run zig pkg is a compatible version for all dependencies recursively, use it.
2. If any tagged releases of zig are compatible with all dependencies recursively, then use the newest one. In this case it will download an official release from ziglang.org, or another mirror, as long as the checksum matches. Of course it will keep a cache and only download if not already cached.
3. If it's a recent master branch build, see if ziglang.org/builds/* has a compatible version for the host system.
4. Finally, fall back to building from source, by downloading a zig-bootstrap tarball and using zig c++ as the system C++ compiler. This allows any version of zig to be built from source. It will be important to communicate a sense of progress and estimated time remaining to the user for this because it could potentially take upwards of 1 hour to do this.

Regardless of which strategy is used, zig pkg should make it clear why the selected strategy has been chosen. For example it would print something like this, "package dependency a.b.c.d depends on zig version range 0.9.0-dev.1700+b644d4936 to 0.9.0-dev.1703+6afcaf4a0 which is not compatible with any tagged releases. Building zig 0.9.0-dev.1703+6afcaf4a0 from source...".

The zig pkg subcommand would have advanced CLI flags which could be used to override the behavior on how to acquire zig versions. It would also have a "--disallow-fetching-compiler-versions" flag, or similar, which would disable the ability to acquire alternate zig versions, emitting an error instead if the version of zig used was not compatible with all the dependencies recursively. Distro maintainers would likely want to always pass this flag when building zig projects. They would then be faced with the chore of getting older packages to add support for the (relatively newer) distro zig version, and newer packages to add support for the (relatively older) distro zig version.

After zig pkg is satisfied with the zig binary that it wants to use, it would call execve on the selected zig version, forwarding all the flags except for the ones having to do with this layer of abstraction, and passing an additional internal flag such as --already-resolved-zig-version. This would cause the zig pkg subcommand to skip this logic and proceed to business as usual.

The most useful command would be zig pkg build. This would be equivalent to zig build except that it would do this zig versioning layer first, possibly invoking a different zig version than the one used to invoke zig pkg. Most Zig projects in their README would have instructions to use zig pkg build rather than zig build, because it means that in practice they would not have to deal with users filing issues for using an incompatible zig version. However they would likely spend a fair amount of time using zig build directly, in order to test with exactly the version of zig that they have in their PATH.

Another subcommand would be zig pkg fetch-zig [version] which would expose the "acquiring a zig compiler version" feature above. For example it would acquire the specified version, and then print to stdout the path to the binary.

Building on top of this would be another subcommand zig pkg up [version] that would update a symlink pointer to the most recent version specified with the up sub-command. For example, assuming that the user runs the following commands:

zig pkg up 0.8.1
zig pkg up 0.9.0
zig pkg up

After this the cache directory would look something like this:

~/.cache/zig/pkg/0.8.1/*
~/.cache/zig/pkg/0.9.0/*
~/.cache/zig/pkg/0.9.0-dev.1703+6afcaf4a0/*
~/.cache/zig/pkg/up -> ~/.cache/zig/pkg/0.9.0-dev.1703+6afcaf4a0/

If the user then ran zig pkg up 0.8.1 then the up symlink would point to the 0.8.1 directory instead. This would allow the user to set up their PATH variable to include ~/.cache/zig/pkg/up and have the zig pkg up command affect the default zig binary.

---

Another motivation for this change is for the IDE experience. Here is a vision that I have:

• Create a new VSCode extension that does both syntax as well as ZLS. Currently there are 2 separate extensions that you have to install. Deprecate those two extensions in favor of the One Extension To Rule Them All.
• This new VSCode extension would have zero configuration. Instead, when the user opens up a project, it would look at the project's declarative information about supported zig versions, and automatically fetch the appropriate zig compiler. It would build the corresponding ZLS version from source, and not ask silly questions such as "where is zig installed?" or "where is zls installed?".

VSCode is just an example; any IDE should be able to accomplish this user experience of simultaneously editing multiple zig different projects that depend on different versions of zig. And obviously the user should not have to install zig manually. If they install an IDE it should do it for them.

[daurnimator]

The intent is that a maintainer would add a min if they discover that their package depends on an issue that has been resolved only after a particular zig version. They would add a max if they discover that a version of zig has done something backwards-incompatible.

• Even if the maintainer does so, they'd be unable to go back and fix the manifest in older commits/releases
• A project may only be incompatible with a zig release for certain targets or if certain optional features are enabled or several other conditional circumstances.

as long as the checksum matches.

Checksum from where?

[nektro]

there's a lot going on in this issue and I have different opinions about each of them.

• +1 to being able to specify a min and/or max zig version for a package
• not a super fan of editor extension controlling where my zig installation is
• edit: also not a huge fan of json since it doesnt have trailing commas and so commits really awkwardly

[matu3ba]

When using the zig package manager (via the zig pkg subcommand), it would detect these fields, and, if found, acquire a suitable Zig compiler version.

Maybe this was discussed or clarified anywhere, but I can not find it for now.
Is the package manager (or at least the fetching/download stuff from internet) a separate (on demand compiled) binary or not?

Assume its the same binary.

Advantage

• only 1 binary instead of 2 => no separate build system etc.
• zig could get away with only shipping the updater like how rustup works.
• more flexibility to put additional package manager features interacting with compiler

Disadvantage

• Branding: a program that downloads and compiles and runs stuff sounds like malware to me on first side.
• sandboxing (preventing internet access) breaks for firejail and bubblewrap or requires local and temporary workarounds every usage (instead of a simple and permanent rule)
• more flexibility ... => Its unclear, what overall parts are necessary. Decoupling sounds more YAGNI to me.

Aside, package manager tend to have alot user CLI flags to customize things or even have a complete plugin system for additional functionality.
Even when one is in a comfortable position to have the build system decluttered (and one can even use it to run analysis tools etc), the following things would be tremendously useful as extensions:

1. https://crates.io/crates/cargo-audit
2. https://crates.io/crates/cargo-cache
3. https://crates.io/crates/cargo-update (users might want to run additional tests/analysis automatically)
4. Tooling for update inspection (ie codecoverage dependent, review of roadmap/purpose changes etc)
5. Tooling for archiving, offline work and searching many distributed repositories (code search engine)

[Deecellar]

I leave my 2 cents, I hate configuration files, we would still need a lock file, but I prefer having a separate step for package managing, since we have a build.zig, having a second file to control the packages makes me kinda sad, but oh well, outside that.

I think json is not fit for this kind of jobs, even if it's a popular one, I prefer either TOML (on the bad side of being compared with rust, but hey, zig is already compared with rust and go) or using a simplified config file.

There is also the option of using a subset of zig or zig directly as a configuration format, which would be fun at the very least, and we have a lot of effort into parsing zig already, downside, is harder to parse for tools not made in zig, and that could detriment places where is not ergonomic to have a zig program (like plugins in certain test).

I always felt that having to write dependencies in text files was kinda overhead, since I had to learn a new structure each new language I went, and reusing the already acquired knowledge for package managing could be beneficial.

Thing is, I have no hard proof here, so I will leave this as my opinion into this. if I can, I wish to make a more insightful comment with things that might be also alternatives, and how they might fit with ergonomics of usage.

[LemonHX]

I feel that simply using Git to manage packages or URLs is not friendly to developers in China, because China has the well-known GFW, and now Chinese developers usually use mirrors. I think there should be at least a way for China and other possible developers. Developers in areas where the network is blocked can create their mirrors.

Go language now has a goproxy option that can be used to configure the behavior of the package manager, but the download of git packages in mainland China is still very slow, so I think there should be at least an official hosting service or something similar to it After git clones first, it is convenient for everyone to manage the mirror in a unified manner?

[Jarred-Sumner]

I love this

Sufficiently popular languages/tooling ends up with an ecosystem of version managers when really it should just be a builtin thing so that by default, developers spend less time thinking about how to upgrade or use the correct version for the software

Both fields would be optional. If min is omitted, it means there is no zig version that is too old. If a max is omitted, it means there is no zig version that is too new. The intent is that a maintainer would add a min if they discover that their package depends on an issue that has been resolved only after a particular zig version.

suggest using a semver range for consistency with the rest of how dependencies are defined (or if not semver, the model you end up with), rather than something one-off

Create a new VSCode extension that does both syntax as well as ZLS. Currently there are 2 separate extensions that you have to install. Deprecate those two extensions in favor of the One Extension To Rule Them All.

This new VSCode extension would have zero configuration. Instead, when the user opens up a project, it would look at the project's declarative information about supported zig versions, and automatically fetch the appropriate zig compiler. It would build the corresponding ZLS version from source, and not ask silly questions such as "where is zig installed?" or "where is zls installed?".

this sounds great

there is planned to be a declarative file format, such as zig-package.json, which sits alongside build.zig and lists information such as the set of packages that are possibly depended on.

I suggest JSON with comments & trailing commas, similar to tsconfig.json. The reason why comments were removed from JSON isn't very compelling. I think JSON overall is good though because (1) schemas are faster than code and (2) so much existing tooling supports JSON-like formats.

For example, Prettier supports auto-formatting jsonc and editor support for JSON schemas mean zig-package.json (or the name you end up with) would have easy support for autocomplete with descriptions and links

Go language now has a goproxy option that can be used to configure the behavior of the package manager, but the download of git packages in mainland China is still very slow, so I think there should be at least an official hosting service or something similar to it After git clones first, it is convenient for everyone to manage the mirror in a unified manner?

One of the nice things about npm is it is easy to mirror. An implicit default URL/host is easier to change and also easier to reference

[tsmanner]

I like the idea of in-built package management, and think that there's some really promising user-land development happening based on zig build already. zig pkg feels like a way to get around the zig build bootstrap problem where a build.zig may itself have external dependencies. My intuition is that adding another layer of tooling isn't truly solving the problem, but rather kicking it down the road. First we have a source file we want to compile, so we have a compiler. Next we have a directory full of them, so we write a build system. Then we realize we have a few directories, so now we write a package manager.

Fundamentally all these things are the same, they take some set of inputs (mostly files and args) and produce a binary. I think nix has done a pretty good job of consolidating that process by merging the build-system and package-manager parts into a single entity that can both describe dependencies within a project, and between projects.

On the topic of configuration formats, I think there are some really promising things coming out of the functional world that do a really good job of providing users the ability to describe complex operations themselves.

Nix's configuration language is the most flexible I've used, but it's tied to nix itself. The nickel project is attempting to provide a generic language that could replace nix-lang in nix while being useful in other contexts, like other build systems.

I keep coming back to the fact that it would be nice to keep everything in build.zig somehow, using something like @marler8997's suggestion from #8025. @tryImport seems like a fairly simple solution, and could pretty easily be applied to a declarative set of package dependencies (probably std.build.Pkg instances, or so) if the @tryImport could take a comptime-known string as an argument. Other assertions could be made, including the zig version stuff, in libraries using that idiom and a build could collect a set of unsatisfied dependencies, do the version constraint solving, and then fetch and/or build them all at once before providing them as libraries to a reinvocation of zig build.

[andrewrk]

I'm convinced that however useful this logic may be, it belongs outside the compiler project itself. This is in the scope of a "zigup"-like project. It is logic that belongs in an IDE extension or a separate CLI program.