Feature wish: support for unsafe reinstall, that does not recompile reverse deps

[lthls]

The motivation: working on the compiler, I regularly want to test my patched compiler on opam packages, which is easy, but when it finds a bug with my compiler, I need to patch again to either fix the bug (if I know what's wrong), or add debugging options to understand better what's happening.

The basic feature I would like in these cases, is some way to reinstall my patched compiler without requiring a rebuild of all the other packages I have installed. (Allowing the reinstall part to be quick is also important, but orthogonal to this problem and should be solved using --assume-built.)

I've already asked for workarounds, and have been suggested the following approaches:

• Use make install by hand in my compiler source tree, assuming it has been configured with the right prefix already. This mostly works, but opam may get confused later by the new versions installed without its knowledge. Also, this does not uninstall the previous version, which is problematic if I only wanted to install bytecode versions of the tools (which are much faster to recompile).
• Generate a .install file, and call opam-installer manually. This means a bit of dirty work in the compiler's Makefiles to generate them properly, and then there's still the issue that for switching from a full build to a bytecode-only build, I would need to have the old .install files around, call opam-installer to remove the old version, then generate the new .install and call again opam-installer to install them.

In the end, for now I don't use the workarounds, in part because recompiling everything is often needed anyway (if my patch changes the cmx files emitted during compilation, for instance) but also because without support for switching from full builds to bytecode-only builds, I need to recompile the whole compiler anyway and this is often the longest part with the current state of flambda2.

But I think having this feature would speed up significantly the debugging phase of compiler development, and could be useful for other people as well.

Cc @gasche, who might be interested in the feature for opam-compiler-conf, and @emillon who I understand wants to know about uses of opam for people working on the compiler.

[lthls]

Update: @AltGr suggested to me that if one pins the ocaml package and edits it so that it only build-depends on ocaml-variants, then one can use opam reinstall ocaml-variants --assume-buit (assuming it was pinned locally with --inplace-build) to remove the current version and reinstall the new one without triggering recompilation of packages which depend on ocaml.
Then if one needs to actually recompile everything (because the format of objects has changed, for instance), opam reinstall ocaml will do the trick.

[gasche]

This sounds interesting; is there a way to make this trick practical to use and document it in ocaml/ocaml/HACKING.adoc? (Editing the ocaml package seems a bit subtle, could we host an ocaml.opam file in the repository instead?)

[lthls]

I can try to write setup steps, make a PR to add them to HACKING.adoc, and you can try them and tell me if they work for you.
I don't know whether hosting ocaml.opam in the compiler's repository is better than giving pinning instructions. If you're confident in our (as compiler developers) ability to maintain an additional ocaml.opam file, then I guess it would work, otherwise relying on the opam repository maintainers for the base file and just adding edit instructions seems likely to require less effort on our part. Maybe @avsm has a more informed opinion on this problem ?

[lthls]

I've run into dependency issues:
ocaml depends on ocaml-variants, this can be turned into a build-only dependency for my purposes.
ocaml-variants post-depends on ocaml, this can be fixed by marking the dependency as build-only.
ocaml depends on ocaml-config, which itself depends on ocaml-variants: I can fix this by marking any of those dependencies as build-only, but I don't know what ocaml-config is supposed to do, so I'm not sure if it makes sense to rerun it after each install or not.
I've been told that @dra27 may be knowledgeable on this, and would welcome some advice.

[avsm]

@lthls The instructions you need for in-place build are in the multicore ocaml's opam-variants file https://github.com/ocaml-multicore/ocaml-multicore/blob/parallel_minor_gc/ocaml-variants.opam

The solutions for opam 2.0 are a bit hit and miss unfortunately; it's difficult to break the dependency deadlock reliably. If you're willing, you can probably install ocaml-config manually -- it sets up various opam variables for the facilities available in that switch.

If you give us a little bit longer, we're looking into supporting this precise in-place usecase properly in opam 2.1; https://github.com/ocaml/opam/wiki/Spec-for-working-with-the-OCaml-compiler

[lthls]

Thanks for the link to the opam file. Has anyone actually used the --assume-built version ? My own experiments and some discussion with @rjbou led me to believe that it was broken since it was introduced.

In the workflow I'm hoping to use, the main point is not to build a switch, but to update the compiler in an existing switch with as little overhead as possible, even if it means using unsafe techniques like lying about dependencies.

I'm not in a hurry here, I just happened to have some time and motivation today for working on this issue. I will of course look again and update my workflow when new opam features become available in a released version.

[avsm]

--assume-built has worked, but it's very brittle (and in my testing, has caused the client to go into infinite loops in some circumstances).

There's one obvious way to update the compiler in-place: to simply use the prefix of the opam-installed version with a local source tree and install into that area. I think the way to go is to wrap all this logic into a simple opam-compiler plugin that hides this gory detail from the CLI. That's a reasonable half-way house to avoid waiting for opam 2.1.0 as well.

[lthls]

I'd like this opam-compiler plugin to be able to uninstall the old compiler before installing the new one. If someone starts working on the plugin, please keep this in mind.

[rjbou]

@lthls @avsm It is not broken since its introduction, but since some opam repo compiler changes I suppose (I didn't tried a bisect on the opam repo :D). My guess is that part of assume-built code was no more working since some ocaml package updates (ocaml-config maybe?), and some dependency calculation graph updates in opam. But it's old stories, it is rewritten (& fixed) in #4211 (and thanks for testing!).

[avsm]

@lthls:

I'd like this opam-compiler plugin to be able to uninstall the old compiler before installing the new one. If someone starts working on the plugin, please keep this in mind.

Could you elaborate on this about why exactly? (this affects the implementation of that feature). It sounds like you want to remove any files installed specifically by the previous compiler installation, but not touch other files installed within the prefix (e.g. by opam packages?).

The other case I'm wondering about is why overwriting isn't sufficient - is it the case that there are often deleted files in the new compiler?

[lthls]

The workflow I'd like to see supported is to first compile a full compiler (with bytecode and native binaries), install it, build some packages, and when one of them fails, recompile the bytecode versions of the compilers only (because it's much faster this way), and only install those.
Then I can install them, and try to recompile to failing package with a fix or more debug info.
The problem is that if the native binaries are still present, the opam package will likely pick them instead of the new bytecode versions, so I'd like opam to remove them for me (at the moment, there isn't any uninstall instruction in the compiler, so opam is the best way to know which files were installed).