--working-dir does not work as expected

[MSoegtropIMC]

Dear Opam Team,

I am trying to build unreleased versions of Coq in a local git repo using opam to ensure it is build in the same way as opam would build it. Unfortunately Coq has some outdated .opam files in its source tree (which of cause can be fixed). So what I do is delete the local opam files and copy proper opam files from the latest release into the local git repo. Since I don't commit these changes I need --working-dir. I also copy the additional files (coq.install and coqide.install).

So what I do is:

cp ~/.opam/repo/default/packages/coq/coq.8.10.2/opam "$COQROOT/coq.opam"
cp ~/.opam/repo/default/packages/coq/coq.8.10.2/files/* "$COQROOT/"
cp ~/.opam/repo/default/packages/coqide/coqide.8.10.2/opam "$COQROOT/coqide.opam"
cp ~/.opam/repo/default/packages/coqide/coqide.8.10.2/files/* "$COQROOT/"
opam install --yes --working-dir "$COQROOT"

Now the interesting thing is that this does work on OSX but it does not work on Ubuntu 18.04. On both I have opam 2.0.5 and OCaml 4.08.1. On Ubuntu when I look into opam's source cache I see the version committed to GIT and not the working dir version. Also on OSX opam seems to be happy when I copy the additonal files into the root folder while on Linux it seems to want them in root/files. I wonder what I am doing wrong.

The output I get on Linux is:

[coq.8.10.2] synchronised from git+file:///home/michael/coq-work-1#issue_11351_version_pick_811
[WARNING] Failed checks in opam file from upstream of coq:
    error 53: Mismatching 'extra-files:' field: "coq.install"
[coq] Conflicting update of the metadata from
git+file:///home/michael/coq-work-1#issue_11351_version_pick_811.
Override files in /home/michael/.opam/4.08.1/.opam-switch/overlay/coq (there
will be a backup)? [Y/n] y
User metadata backed up in /home/michael/.opam/backup/coq.bak
[WARNING] Ignoring file
          /home/michael/.opam/4.08.1/.opam-switch/sources/coq/files/coq.install
          with invalid hash
[coqide] Conflicting update of the metadata from
git+file:///home/michael/coq-work-1#issue_11351_version_pick_811.
Override files in /home/michael/.opam/4.08.1/.opam-switch/overlay/coqide (there
will be a backup)? [Y/n] y
User metadata backed up in /home/michael/.opam/backup/coqide.bak
[WARNING] Ignoring file
          /home/michael/.opam/4.08.1/.opam-switch/sources/coqide/files/coqide.install
          with invalid hash
The following actions will be performed:
  ∗ install coq    8.10.2*
  ∗ install coqide 8.10.2*
===== ∗ 2 =====
[coq.8.10.2] synchronised from git+file:///home/michael/coq-work-1#issue_11351_version_pick_811
[coqide.8.10.2] synchronised from git+file:///home/michael/coq-work-1#issue_11351_version_pick_811

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>

#=== ERROR while compiling coq.8.10.2 =========================================#
opam: "open" failed on /home/michael/.opam/4.08.1/.opam-switch/sources/coq/files/coq.install: No such file or directory


<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
┌─ The following actions failed
│ λ build coq 8.10.2
└─ 
╶─ No changes have been performed

Best regards,

Michael

[rjbou]

Thanks for the report!

On Ubuntu when I look into opam's source cache I see the version committed to GIT and not the working dir version.

You shouldn't have different behavior according the os... working dir uses rsync & git, that are not different on those os.

Also on OSX opam seems to be happy when I copy the additonal files into the root folder while on Linux it seems to want them in root/files.

You can add them on the root or file folder, opam should take them both. In this issue (and especially the gist) you can find information about this kind of actions.

From you log, it seems that there is a mismatching checksum between the found file and the checksum in the opam file. Can you run using on verbose mode-vv to have check command output and see if the command fails on the check or another thing. it will also give us information on what files are copied.

Note that working dir had some fixes/updates in opam' master.

[MSoegtropIMC]

The checksum mismatch is because the opam files have been replaced with the version from git, which have a different checksum. The extra files themselves don't exist in git (don't ask ...), so they are not updated. I attached the verbose log.

Thanks for the links on more detailed information.

OpemVerboseLog.txt

[rjbou]

Can you check that the problem is resolved in master (or in next 2.1.0~beta)? there was several fixes on working-dir merged.
From you log, it seems that coq.opam & coqide.opam files has well been rsync-copied, so checksum shouldn't mismatch.

[rjbou]

Any news @MSoegtropIMC ?

[MSoegtropIMC]

I hesitaed to update opam. One question: if I update opam, do I need to recompile everything or will it take my switches as it finds them?

[rjbou]

Today's master will not update your switches, but tomorrow one could :D. In any case, it is supposed to be completely transparent, without switch rebuild, and break free.
If you want to be sure to be able to retrieve you old configuration/switches, you can simply save a copy of your current opamroot .opam directory and in the end, set it back . Note that local switches (directory switches) won't be saved in you opamroot backup.

[MSoegtropIMC]

Indeed, I can just backup .opam. I will try it tomorrow.

[rjbou]

Any news on this one ?

[MSoegtropIMC]

Sorry I didn't do any local repo opam stuff the last weeks, but I need to create a few packages this week, which will require testing this. So I will find out soon.

[dra27]

This should hopefully have been addressed during the 2.1.0 development cycle - please re-open if not!