opam-state 2.0.8 and opam 2.0.8 are competing to rebuild the repository cache

[samoht]

Reproduction case:

$ opam --version
2.0.8
$ export OPAMDEBUG=1
$ opam install opam-0install opam-state.2.0.8
[..]
$ opam 0install dune
00:00.004  SYSTEM                 mkdir /tmp/opam-xxx-33243
00:00.000  GSTATE                 LOAD-GLOBAL-STATE @ /home/opam/.opam
00:00.000  RSTATE                 LOAD-REPOSITORY-STATE @ /home/opam/.opam
00:00.001  RSTATE                 Bad cache: incompatible magic string "18F028B2" (expected "3C5DCD19").
00:00.002  RSTATE                 Invalid cache, removing
00:00.002  SYSTEM                 rm /home/opam/.opam/repo/state.cache
00:00.006  RSTATE                 No cache found
[...]
$ opam info dune
00:00.001  GSTATE                 LOAD-GLOBAL-STATE @ /home/opam/.opam
00:00.002  RSTATE                 LOAD-REPOSITORY-STATE @ /home/opam/.opam
00:00.004  RSTATE                 Bad cache: incompatible magic string "3C5DCD19" (expected "18F028B2").
00:00.004  RSTATE                 Invalid cache, removing
00:00.004  SYSTEM                 rm /home/opam/.opam/repo/state.cache
00:00.009  RSTATE                 No cache found
[...]

This means that every invocation of tools linked with opam-state takes a 20s hit on startup to rebuild these cache.

[AltGr]

Thanks for reporting; actually hit this, but since I'm always on dev versions, I assumed that wasn't a general issue...

The issue is that, to avoid permanent recompilations of opam, the reference OpamVersion.gitversion is initialised to None in opam-core, then initialised at runtime in OpamCliMain.run, using a custom module containing the git hash that dune generates and links as late as possible (i.e. in opam-client).

Now, this implies that OpamVersion.full () returns different results when running from a binary that both includes opam-client and ran the proper initialisation, and when running from a lib that didn't meet these conditions. And this value is used to generate the magic strings that determine the validity of the cache, as shown above.

After pondering the different solutions (e.g. strip the git-hashes from releases ?), I think the best course of action might be to re-add the generated OpamGitVersion module to OpamCore, but only when building releases or opam packages, restoring consistency. While making sure to keep the current behaviour otherwise, in order to avoid recompiling everything at every commit though.

[dra27]

Dev meeting proposal for both 2.0.x and 2.1.x - in addition, the name of the state cache can include the magic number, which means that different library versions would create different states. This would mean that, for example, two different dev releases would not keep erasing each other's state cache. At opam update (or opam repo add, etc. - i.e. any stage where the state caches are erased) all state caches would be blown away.