Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use new hidden deps Merlin support for handling implicit-transitive-deps false #10535

Merged
merged 12 commits into from
Oct 11, 2024

Conversation

voodoos
Copy link
Collaborator

@voodoos voodoos commented May 15, 2024

With release 5.2 the compiler introduces a new flag -H to introduce dependencies that are required for linking but should be hidden from the currently built module. Merlin for 5.2 follows with two new configuration directives: BH and SH that allows to provide hidden build and source paths.

This PR effectively takes advantage of these new directives to correctly communicate hidden dependencies to Merlin.

All versions of Merlin since 4.6 will accept that flag, but only the preview versions for 5.2 can actually take advantage of this additional information.

@nojb
Copy link
Collaborator

nojb commented May 17, 2024

Hello @voodoos, thanks for this PR. Note that #9333 is being worked on at the moment (cc @MA0010). Once that is done, you will be able to simplify the patch because the lists of requires given by Compilation_context.requires will be annotated with a flag indicating whether each dependency is direct or transitive (= hidden), and you won't need to do this computation "just" for Merlin. Moreover, I think it would be great if we could merge this PR at the same time that Dune itself is able to support -H.

Accordingly, I suggest to suspend this PR until #9333 is done; does that sound OK? I hope that there will be a PR for #9333 in the next week or so.

@nojb nojb self-assigned this May 17, 2024
@nojb
Copy link
Collaborator

nojb commented Jun 28, 2024

@voodoos #10644 has been merged; so it should now be possible to finalize this PR by using Compilation_context.requires_hidden. Thanks for waiting!

@voodoos voodoos force-pushed the merlin-hidden-deps branch from e2531ee to bcccf15 Compare July 1, 2024 13:35
@Leonidas-from-XIV
Copy link
Collaborator

Hi @voodoos could you rebase the PR? We were talking about this in the last dune-dev meeting and it would be nice to have it as part of the 3.17 release.

@voodoos voodoos force-pushed the merlin-hidden-deps branch from bcccf15 to 257a373 Compare October 4, 2024 16:11
@voodoos voodoos force-pushed the merlin-hidden-deps branch from a5b1ea6 to e26d0ce Compare October 7, 2024 12:13
@voodoos
Copy link
Collaborator Author

voodoos commented Oct 7, 2024

@Leonidas-from-XIV rebased and signed-off

@voodoos voodoos force-pushed the merlin-hidden-deps branch from 7465100 to b3cc73e Compare October 7, 2024 14:55
Copy link
Collaborator

@Leonidas-from-XIV Leonidas-from-XIV left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please add a changelog entry? This is one of the highlights of the release, would be good for people to be aware of it :)

src/dune_rules/merlin/merlin.ml Outdated Show resolved Hide resolved
@voodoos voodoos force-pushed the merlin-hidden-deps branch from 3531597 to b4641d9 Compare October 9, 2024 16:24
Use new BH / SH directives

Make specific tests for OCaml 5.2

Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>
Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>
Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>
Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>
Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>
Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>
Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>
Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>
@voodoos voodoos force-pushed the merlin-hidden-deps branch from b4641d9 to 775915d Compare October 10, 2024 14:21
test/blackbox-tests/test-cases/overlapping-deps.t Outdated Show resolved Hide resolved
-> required by _build/default/proj2/.merlin-conf/lib-bar
-> required by _build/default/proj2/bar.cma
-> required by %{cma:proj2/bar} at command line:1
[1]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting that this triggers the error now, this could be a regression for the release but it seems like the error is what should've happened here anyway.

@@ -96,8 +96,7 @@ that wasn't found:
1 | (executable (name prog) (libraries a))
^
Error: Library "a" not found.
-> required by _build/default/c/.prog.eobjs/byte/dune__exe__Prog.cmi
-> required by _build/default/c/.prog.eobjs/native/dune__exe__Prog.cmx
-> required by _build/default/c/.merlin-conf/exe-prog
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My main concern is that this changes so many tests, but given the dependency that is getting reported is an internal one and stays an internal one I believe this is not an issue.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like the PR changes some evaluation order that makes Dune errors after following a different branch. Not sure what causes this however...

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rgrinberg Can you please weight in whether this is an issue? It doesn't seem to be wrong, but maybe the order is important in some way I am not aware of?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found the difference: before we only "peeked" (with Resolve.peek) at the dependencies and now we "read" them.

I will revert that change as is seems wrong to introduce dependencies here.

voodoos and others added 2 commits October 10, 2024 16:36
Co-authored-by: Marek Kubica <marek@xivilization.net>
Signed-off-by: Ulysse <5031221+voodoos@users.noreply.github.com>
Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>
@rgrinberg
Copy link
Member

@nojb I think you've assigned yourself to this PR. Do you mind taking a final look, and if everything is OK, merging?

@@ -0,0 +1,2 @@
- Use Merlin new configuration directives `BH` and `SH` for hidden dependencies.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would mention that hidden dependencies correspond to the -h flag instead of internal merlin directives which most people are unaware of.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, it is always better to write Changes entries from the point of view of the user, not the developer. In this case, something like

- Make Merlin/OCaml-LSP aware of "hidden" dependencies used by
  `(implicit_transitive_deps false)` via the `-H` compiler flag.

would be more useful for the occasional reader.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took the liberty of tweaking the Changes entry in this manner. Planning to merge once CI passes. Thanks!

Copy link
Collaborator

@nojb nojb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@@ -0,0 +1,2 @@
- Use Merlin new configuration directives `BH` and `SH` for hidden dependencies.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, it is always better to write Changes entries from the point of view of the user, not the developer. In this case, something like

- Make Merlin/OCaml-LSP aware of "hidden" dependencies used by
  `(implicit_transitive_deps false)` via the `-H` compiler flag.

would be more useful for the occasional reader.

nojb added 2 commits October 11, 2024 06:30
Signed-off-by: Nicolás Ojeda Bär <n.oje.bar@gmail.com>
@nojb nojb merged commit b39ad6d into ocaml:main Oct 11, 2024
26 of 27 checks passed
@voodoos
Copy link
Collaborator Author

voodoos commented Oct 11, 2024

Thank you @nojb !

anmonteiro pushed a commit to anmonteiro/dune that referenced this pull request Nov 17, 2024
…eps false (ocaml#10535)

Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>
maiste added a commit to maiste/opam-repository that referenced this pull request Nov 25, 2024
CHANGES:

### Fixed

- Show the context name for errors happening in non-default contexts.
  (ocaml/dune#10414, fixes ocaml/dune#10378, @jchavarri)

- Correctly declare dependencies of indexes so that they are rebuilt when
  needed. (ocaml/dune#10623, @voodoos)

- Don't depend on coq-stdlib being installed when expanding variables
  of the `coq.version` family (ocaml/dune#10631, fixes ocaml/dune#10629, @gares)

- Error out if no files are found when using `copy_files`. (ocaml/dune#10649, @jchavarri)

- Re_export dune-section private library in the dune-site library stanza,
  in order to avoid failure when generating and building sites modules
  with implicit_transitive_deps = false. (ocaml/dune#10650, fixes ocaml/dune#9661, @MA0100)

- Expect test fixes: support multiple modes and fix dependencies when there is
  a custom runner (ocaml/dune#10671, @vouillon)

- In a `(library)` stanza with `(extra_objects)` and `(foreign_stubs)`, avoid
  double linking the extra object files in the final executable.
  (ocaml/dune#10783, fixes ocaml/dune#10785, @nojb)

- Map `(re_export)` library dependencies to the `exports` field in `META` files,
  and vice-versa. This field was proposed in to
  https://discuss.ocaml.org/t/proposal-a-new-exports-field-in-findlib-meta-files/13947.
  The field is included in Dune-generated `META` files only when the Dune lang
  version is >= 3.17.
  (ocaml/dune#10831, fixes ocaml/dune#10830, @nojb)

- Fix staged pps preprocessors on Windows (which were not working at all
  previously) (ocaml/dune#10869, fixes ocaml/dune#10867, @nojb)

- Fix `dune describe` when an executable is disabled with `enabled_if`.
  (ocaml/dune#10881, fixes ocaml/dune#10779, @moyodiallo)

- Fix an issue where C stubs would be rebuilt whenever the stderr of Dune was
  redirected. (ocaml/dune#10883, fixes ocaml/dune#10882, @nojb)

- Format long lists in s-expressions to fill the line instead of formatting
  them in a vertical way (ocaml/dune#10892, fixes ocaml/dune#10860, @nojb)

- Fix the URL opened by the command `dune ocaml doc`. (ocaml/dune#10897, @gridbugs)

- Fix the file referred to in the error/warning message displayed due to the
  dune configuration version not supporting a particular configuration
  stanza in use. (ocaml/dune#10923, @H-ANSEN)

- Fix `enabled_if` when it uses `env` variable. (ocaml/dune#10936, fixes ocaml/dune#10905, @moyodiallo)

- Fix exec -w for relative paths with --root argument (ocaml/dune#10982, @gridbugs)

- Do not ignore the `(locks ..)` field in the `test` and `tests` stanza
  (ocaml/dune#11081, @rgrinberg)

- Tolerate files without extension when generating merlin rules.
  (ocaml/dune#11128, @anmonteiro)

### Added

- Make Merlin/OCaml-LSP aware of "hidden" dependencies used by
  `(implicit_transitive_deps false)` via the `-H` compiler flag. (ocaml/dune#10535, @voodoos)

- Add support for the -H flag (introduced in OCaml compiler 5.2) in dune
  (requires lang versions 3.17). This adaptation gives
  the correct semantics for `(implicit_transitive_deps false)`.
  (ocaml/dune#10644, fixes ocaml/dune#9333, ocsigen/tyxml#274, ocaml/dune#2733, ocaml/dune#4963, @MA0100)

- Add support for specifying Gitlab organization repositories in `source`
  stanzas (ocaml/dune#10766, fixes ocaml/dune#6723, @H-ANSEN)

- New option to control jsoo sourcemap generation in env and executable stanza
  (ocaml/dune#10777, fixes ocaml/dune#10673, @hhugo)

- One can now control jsoo compilation_mode inside an executable stanza
  (ocaml/dune#10777, fixes ocaml/dune#10673, @hhugo)

- Add support for specifying default values of the `authors`, `maintainers`, and
  `license` stanzas of the `dune-project` file via the dune config file. Default
  values are set using the `(project_defaults)` stanza (ocaml/dune#10835, @H-ANSEN)

- Add names to source tree events in performance traces (ocaml/dune#10884, @jchavarri)

- Add `codeberg` as an option for defining project sources in dune-project
  files. For example, `(source (codeberg user/repo))`. (ocaml/dune#10904, @nlordell)

- `dune runtest` can now run individual tests with `dune runtest mytest.t`
  (ocaml/dune#11041, @Alizter).

- Wasm_of_ocaml support (ocaml/dune#11093, @vouillon)

- Add a `coqdep_flags` field to the `coq` field of the `env` stanza, and to the `coq.theory` stanza, allowing to configure `coqdep` flags.
  (ocaml/dune#11094, @rlepigre)

### Changed

- Remove all remnants of the experimental `patch-back-source-tree`. (ocaml/dune#10771,
  @rgrinberg)

- Change the preset value for author and maintainer fields in the
  `dune-project` file to encourage including emails. (ocaml/dune#10848, @punchagan)

- Tweak the preset value for tags in the `dune-project` file to hint at topics
  not having a special meaning. (ocaml/dune#10849, @punchagan)

- Change some colors to improve readability in light-mode terminals
  (ocaml/dune#10890, @gridbugs)

- Forward the linkall flag to jsoo in whole program compilation as well (ocaml/dune#10935, @hhugo)

- Configurator uses `pkgconf` as pkg-config implementation when available
  and forwards it the `target` of `ocamlc -config`. (ocaml/dune#10937, @pirbo)

- Enable Dune cache by default. Add a new Dune cache setting
  `enabled-except-user-rules`, which enables the Dune cache, but excludes
  user-written rules from it. This is a conservative choice that can avoid
  breaking rules whose dependencies are not correctly specified. This is the
  current default. (ocaml/dune#10944, ocaml/dune#10710, @nojb, @ElectreAAS)

- Do not add `dune` dependency in `dune-project` when creating projects with
  `dune init proj`. The Dune dependency is implicitely added when generating
  opam files (ocaml/dune#11129, @Leonidas-from-XIV)
maiste added a commit to maiste/opam-repository that referenced this pull request Nov 27, 2024
CHANGES:

### Fixed

- Show the context name for errors happening in non-default contexts.
  (ocaml/dune#10414, fixes ocaml/dune#10378, @jchavarri)

- Correctly declare dependencies of indexes so that they are rebuilt when
  needed. (ocaml/dune#10623, @voodoos)

- Don't depend on coq-stdlib being installed when expanding variables
  of the `coq.version` family (ocaml/dune#10631, fixes ocaml/dune#10629, @gares)

- Error out if no files are found when using `copy_files`. (ocaml/dune#10649, @jchavarri)

- Re_export dune-section private library in the dune-site library stanza,
  in order to avoid failure when generating and building sites modules
  with implicit_transitive_deps = false. (ocaml/dune#10650, fixes ocaml/dune#9661, @MA0100)

- Expect test fixes: support multiple modes and fix dependencies when there is
  a custom runner (ocaml/dune#10671, @vouillon)

- In a `(library)` stanza with `(extra_objects)` and `(foreign_stubs)`, avoid
  double linking the extra object files in the final executable.
  (ocaml/dune#10783, fixes ocaml/dune#10785, @nojb)

- Map `(re_export)` library dependencies to the `exports` field in `META` files,
  and vice-versa. This field was proposed in to
  https://discuss.ocaml.org/t/proposal-a-new-exports-field-in-findlib-meta-files/13947.
  The field is included in Dune-generated `META` files only when the Dune lang
  version is >= 3.17.
  (ocaml/dune#10831, fixes ocaml/dune#10830, @nojb)

- Fix staged pps preprocessors on Windows (which were not working at all
  previously) (ocaml/dune#10869, fixes ocaml/dune#10867, @nojb)

- Fix `dune describe` when an executable is disabled with `enabled_if`.
  (ocaml/dune#10881, fixes ocaml/dune#10779, @moyodiallo)

- Fix an issue where C stubs would be rebuilt whenever the stderr of Dune was
  redirected. (ocaml/dune#10883, fixes ocaml/dune#10882, @nojb)

- Fix the URL opened by the command `dune ocaml doc`. (ocaml/dune#10897, @gridbugs)

- Fix the file referred to in the error/warning message displayed due to the
  dune configuration version not supporting a particular configuration
  stanza in use. (ocaml/dune#10923, @H-ANSEN)

- Fix `enabled_if` when it uses `env` variable. (ocaml/dune#10936, fixes ocaml/dune#10905, @moyodiallo)

- Fix exec -w for relative paths with --root argument (ocaml/dune#10982, @gridbugs)

- Do not ignore the `(locks ..)` field in the `test` and `tests` stanza
  (ocaml/dune#11081, @rgrinberg)

- Tolerate files without extension when generating merlin rules.
  (ocaml/dune#11128, @anmonteiro)

### Added

- Make Merlin/OCaml-LSP aware of "hidden" dependencies used by
  `(implicit_transitive_deps false)` via the `-H` compiler flag. (ocaml/dune#10535, @voodoos)

- Add support for the -H flag (introduced in OCaml compiler 5.2) in dune
  (requires lang versions 3.17). This adaptation gives
  the correct semantics for `(implicit_transitive_deps false)`.
  (ocaml/dune#10644, fixes ocaml/dune#9333, ocsigen/tyxml#274, ocaml/dune#2733, ocaml/dune#4963, @MA0100)

- Add support for specifying Gitlab organization repositories in `source`
  stanzas (ocaml/dune#10766, fixes ocaml/dune#6723, @H-ANSEN)

- New option to control jsoo sourcemap generation in env and executable stanza
  (ocaml/dune#10777, fixes ocaml/dune#10673, @hhugo)

- One can now control jsoo compilation_mode inside an executable stanza
  (ocaml/dune#10777, fixes ocaml/dune#10673, @hhugo)

- Add support for specifying default values of the `authors`, `maintainers`, and
  `license` stanzas of the `dune-project` file via the dune config file. Default
  values are set using the `(project_defaults)` stanza (ocaml/dune#10835, @H-ANSEN)

- Add names to source tree events in performance traces (ocaml/dune#10884, @jchavarri)

- Add `codeberg` as an option for defining project sources in dune-project
  files. For example, `(source (codeberg user/repo))`. (ocaml/dune#10904, @nlordell)

- `dune runtest` can now run individual tests with `dune runtest mytest.t`
  (ocaml/dune#11041, @Alizter).

- Wasm_of_ocaml support (ocaml/dune#11093, @vouillon)

- Add a `coqdep_flags` field to the `coq` field of the `env` stanza, and to the `coq.theory` stanza, allowing to configure `coqdep` flags.
  (ocaml/dune#11094, @rlepigre)

### Changed

- Remove all remnants of the experimental `patch-back-source-tree`. (ocaml/dune#10771,
  @rgrinberg)

- Change the preset value for author and maintainer fields in the
  `dune-project` file to encourage including emails. (ocaml/dune#10848, @punchagan)

- Tweak the preset value for tags in the `dune-project` file to hint at topics
  not having a special meaning. (ocaml/dune#10849, @punchagan)

- Change some colors to improve readability in light-mode terminals
  (ocaml/dune#10890, @gridbugs)

- Forward the linkall flag to jsoo in whole program compilation as well (ocaml/dune#10935, @hhugo)

- Configurator uses `pkgconf` as pkg-config implementation when available
  and forwards it the `target` of `ocamlc -config`. (ocaml/dune#10937, @pirbo)

- Enable Dune cache by default. Add a new Dune cache setting
  `enabled-except-user-rules`, which enables the Dune cache, but excludes
  user-written rules from it. This is a conservative choice that can avoid
  breaking rules whose dependencies are not correctly specified. This is the
  current default. (ocaml/dune#10944, ocaml/dune#10710, @nojb, @ElectreAAS)

- Do not add `dune` dependency in `dune-project` when creating projects with
  `dune init proj`. The Dune dependency is implicitely added when generating
  opam files (ocaml/dune#11129, @Leonidas-from-XIV)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants