A better depext

[AltGr]

Context: see this original design document, and the depext plugin.

This supersedes and closes issue #2172.

Current status

While we have an implementation of the plugin that has been tested widely for a while, in particular by the CI systems around, it's not included into opam (which means beginners might hit missing depexts; opam tries to be helpful and advises to use it on failures, but it's still one more step). Also, the semantics is a bit weird, and it doesn't support versions of OSes or distributions.

1. The current field depexts: is documented in the manual. It associates sets of tags to lists of package names.

2. The depext plugin analyses the host system to deduce:

• the package manager to use, with pre-defined commands for checking that a set of package is installed, and installing them
• a set of pre-defined tags based on arch, OS, distribution

For each package, the depexts: bindings are searched for keys which are a subset of the inferred tags for the host, and the associated system package names are added to the requirements, to be handled by the system-specific package manager.

Limitations

The syntax is a bit heavy, leading to redundant specification, typically for debian and ubuntu:

depexts: [
  [["debian"] ["libssl-dev"]]
  [["ubuntu"] ["libssl-dev"]]
  [["centos"] ["openssl-devel"]]
]

Also, this doesn't handle any notion of versioning (#2168), with only static tags, so while adding tags for each distro/OS version could be possible, that doesn't seem realistic.

A possible new syntax

Filters and variables are already used throughout in opam files, are more expressive and flexible, and have easier to understand semantics. Instead of the pre-defined tags, we could infer a set of host-describing variables (we already have arch, os) ; either hardcoded (using the code from opam-depext), or even using the new eval-variables: field from the main config file, that (lazily) defines variables from shell commands. This proposed patch by @dra27 goes in that direction.

Then a possible syntax, closer to what the other fields use, would be:

depexts: [
  ["libssl-dev"] { distro = "debian" | distro = "ubuntu" }
  ["openssl-devel"] { distro = "centos" & distro-version >= "7.2" }
]

we also benefit from version ordering as defined in the general opam format.

This still doesn't allow specifying versions of system packages, but handling that across packaging systems doesn't seem easily doable. Besides, we are the only ones to provide many versions for a single given package aren't we ? Eg Linux distributions have one version per package per release.

Remains the question of unavailable depexts: in the example above, what would happen if distro-version was less than "7.2" ? As written above, the depext would simply be dropped (and the package then fail to install, without a clear explanation). We could help, if these new variables are also available in the available: field, by adding:

available: [ !(distro = "centos" & distro-version < "7.2") ]

but that seems a little redundant. Another possibility would be to further extend the format proposed above, with for example triples <sys-packages> <condition> <validation>, where if <condition> holds <packages> are added, but <validation> must hold as well or it's a failure. E.g.

["openssl-devel"] { distro = "centos" } { distro-version >= "7.2" }

[AltGr]

I didn't give details about integrating the depext plugin into opam; the original design document describes that aspect, and the code is already present in the plugin, so it's not the biggest issue.

At the moment, except for opam list --external, opam itself has no understanding of the meaning of the depexts: contents. I think this should be resolved first before we speak about integration.

[AltGr]

Another upside of using variables is that the user can override them if needed.

[avsm]

Another thing that'd be useful is to somehow specify OS-specific remotes -- for instance, the LLVM packages needs a PPA to be added on Ubuntu before the package names make any sense. It's quite easy to add a PPA in as part of the depext invocation, but the syntax for doing so in the OPAM file is far less obvious...

[AltGr]

maybe a syntax "repository:package" or "package@repository", which the system-package-manger-handling code instance for Ubuntu would know how to make sense of ?

[hannesm]

FWIW I'd appreciate depext to not become an integral part of opam (and actually consider depext to be out of scope for opam - it violates the policy of least surprise). I usually fake install depext (I don't have sudo (either installed or permissions) on most systems, and want to explicitly install system packages).

[dbuenzli]

(and actually consider depext to be out of scope for opam - it violates the policy of least surprise)

There's no least surprise being violated you have to invoke them explicitly. Besides having a --dry-run option to it would even tell you what you need to do that you can perform manually if you wish so. I think depext is fully in scope for opam and actually greatly enhances the usability of the package universe.

[avsm]

@hannesm my main driver for integrating depext is so that we can have genuinely coinstallable packages. Right now a bulk build fails at the depext phase for some operating systems (e.g. mariadb/mysql on most things other than Nixos).

However, I do agree with the sentiment that it should be easier to independently update the opam-depext library itself from the main opam release. It's been very nice having a lightweight plugin that can be released via the plugin mechanism. I'm not sure what the best way to do this though...

[hannesm]

@dbuenzli opam calling sudo and the system package manager is surprising, at least for me. I stated I'd be concerned if depext becomes an integral part of opam. Having opam telling me "please install package X" is for sure useful information.

[dbuenzli]

@dbuenzli opam calling sudo and the system package manager is surprising, at least for me.

Again there's no surprise: it will never do this on opam install and you know it may on opam depext.

[hannesm]

@avsm I fail to see how integrating depext into opam will let you install both mariadb and mysql at the same time on an arbitrary operating system.

[dra27]

@hannesm - coming from an OS world where virtually everything is done for you, it's very surprising to me to be told to do something by a program rather than it simply offering to do it! However, I agree that if opam depext becomes properly integrated (something I'm very much after too, for the different reason that Windows opam is pretty rubbish without a depext solution, since it lacks an OS package mangler) that perhaps one of the prominent questions at opam init (or another configurable option) should be whether opam automatically runs depext commands.

[avsm]

@avsm I fail to see how integrating depext into opam will let you install both mariadb and mysql at the same time on an arbitrary operating system.

@hannesm That's the point -- right now opam-depext will happily try to coinstall both and fail at the OS package manager. With the depext constraints integrated, the solver will not even select both simultaneously at the opam package level.

This in turn greatly simplifies how we can leverage the solver to do bulk doc builds for docs.ocaml.org and similar setups.

[AltGr]

About system package conflicts

Even if we "integrate" depext, I don't see how we could lift system PM conflicts into the opam solver; at the best, what we could do is avoid situations like opam install mysql (* gets the depexts and installs *); opam install mariadb (* gets the depexts, sys installs and removes mysql; opam doesn't realise and thinks this is ok *). If we attempt to install both at once, opam will still attempt to install, and the host PM will fail.

In cases like this, I don't see a way around having two conf-* packages and manually marking them as conflicting (possibly with OS-specific filters. These aren't allowed in conflicts at the moment, only in depends, but that could be added).

About integrating depexts

It certainly won't become a requirement of opam to be root on one's machine. Even in the non-root case, it could be useful to use the host PM to detect packages that are not available based on their depexts (get the list of host installed packages once, then use it to filter packages with depexts).

That's a bit more complex, because there would be two possible modes, even before the solver is called: one that is guaranteed not to ask for depexts, but might fail with package X can't be installed because it requires host packages Y that are absent, if no solution can avoid the depexts ; and another that solves normally, and then prompts to install the missing packages using the OS, supplying the commands and/or asking and sudo-ing to run them.

[avsm]

@AltGr

If we attempt to install both at once, opam will still attempt to install, and the host PM will fail.

Ah, can't the OS constraints be propagated to aspcud as well? I'm thinking about the trick of using the solver to get a list of co-installable bulk build packages, and it would be nice if the solver could calculate a set of packages that are both compatible at an OCaml level as well as a depext level (i.e. with the end result that a opam install all-the-things would actually succeed)

[AltGr]

For the specific case of Debian, that should indeed be possible. But I really don't think we can find a reliable way to extract conflicts and coinstallability from every package management system out there, unless we want to iteratively check every possible pair with --dry-run (and even that might not be completely reliable).
Like we manually document the depexts, I think we should rely on specifying the conflicts between conf-* packages.

Note that opam 2 has a feature that makes it possible for opam to detect that some OS feature a conf-* package relied on has been removed or changed, and mark the package appropriately.

[avsm]

Ahh, I was thinking of manually writing the OS conflicts in the opam files, not automatically extracting them from the OS package manager. I'd be happy if the constraint language in OPAM2 were expressive enough to make this possible (by capturing the OS distro, arch, and version number in variables).

[AltGr]

Ah! In that case it's doable indeed:

• defining OS variables: already discussed above, we are going in that direction (pre-defined os and arch are here, can add more, either hardcoded or through the eval-variables: config field)
• allowing filters in conflicts: they are now allowed in depends and depopts, but not in conflicts atm. This can be added

[AltGr]

That would result in e.g. conf-mariadb having:

conflicts: [ "conf-mysql" { os != "nixos" } ]

[avsm]

That sounds right. Can you think of a scheme whereby these variables could be detected independently of the OPAM binary, so that we can release support for new depexts independently of OPAM (i.e. the current scheme of opam-depext plugins)

[AltGr]

Well, eval-variables: can technically be changed independently of opam, but maybe not in a convenient way. It is in the built-in, default configuration file used for opam init, but can also be picked up from /etc/opamrc, ~/.opamrc, or a specified conf file. So it can be overriden for opam init, but with a specific setup only. (Note that I just added opam init --config URL, which could be convenient here, though, although there are security concerns with it)

The bigger issue might be that there is no mechanism to easily update it on an existing opam root, at the moment: you would need to go edit the field in ~/.opam/config. Possible fixes include:

• a command to edit configuration (this is at the design stage, we have opam config set-global but it's only for static variables)
• a command to "absorb" a new conf file on an existing opam root (might be just opam init --conf)
• a generic tool to surgically edit any opam-syntax file (could be done as a simple wrapper over opam-file-format (maybe opam-format) and would be more generally useful to avoid some broken sed invocations).

[dbuenzli]

One thing that came to mind today is that depexts as they exist provide an information bridge between opam and external packages for opam install, it would be nice if they also did so on opam update to detect changes in the packages whose dependency was needed on install.

[bluddy]

Would it perhaps make sense to make a sort-of second opam-repository for depexts? (A depext-repository?) Each package would specify its name under each package manager, as well as its conflicts. It would free up OCaml packages from having to specify this information (and possibly getting it wrong) in the opam file. It also deals with the issue of duplication of information between different opam packages requiring the same depext.

Of course, the universe of this depext-repository would start out very small and only grow as needed by OCaml packages. At the same time, this sort of information would be very useful outside of the OCaml world as well.

[bluddy]

Answering myself, I guess that's what the conf-X packages already are. Scratch that.

[AltGr]

Cross-linking #3058, which is an important step for this