Directories whose names start with underscore get ignored for foreign build sandboxing

[grievejia]

Expected Behavior

I'm trying to build a 3rd-party C library as part of my project using foreign build sandboxing.

I closely followed the instructions in the doc, but found out that my build failed because it cannot find files under any subdirectories whose name starts with _. Looking into the _build directory, I can confirm that files under those directories were just non-existent in the first place.

I understand the need to ignore those dirs when building OCaml sources. But dropping them for foreign libraries do not make too much sense for me.

Is there a good way to work around this problem at the moment? Ideally I would like to avoid making any changes to any pathnames of the 3rd-party library itself.

Actual Behavior

I expect all source files, regardless of their path names, get copied into the _build directory

Reproduction

Here's a small repro:

> ls
_build       _opam        dune-project test

> cat dune_project
(lang dune 2.9)

> tree test
test
└── lib
    ├── dune
    └── vendor
        ├── _b
        │   └── b.c
        └── a
            └── a.c

> cat test/lib/dune
(data_only_dirs vendor)

(rule
 (deps (source_tree vendor))
 (targets b.c)
 (action
 (no-infer
  (progn 
   (copy vendor/_b/b.c b.c)
   ))))

(library
 (name test))

> dune build
Error: _build/default/test/lib/vendor/_b/b.c: No such file or directory

# Note that if I change the source of the `copy` stanza into `vendor/a/a.c`, the build would succeed.

In my real setup, the actions would be more complicated (e.g. cd vendor && ./configure && make foo && cp libfoo.a ../libfoo.a). But this small example should be sufficient to demonstrate the crux of the problem here.

Specifications

• Version of dune (output of dune --version): 2.9.0
• Version of ocaml (output of ocamlc --version): 4.12.0
• Operating system (distribution and version): macOS Big Sur

Additional information

• Link to gist with verbose output (run dune with the --verbose flag): https://gist.github.com/grievejia/bec0c7c4fdd0546793aca817659d320f

[nojb]

dune has a stanza to control which directories are ignored, (dirs ...). You can un-ignore directories such as _b by using (dirs :standard _b). However, this stanza applies only to immediate subdirectories which complicates your case where _b is not an immediate child of the directory containing the closest dune file.

To make it work in your example, you can add the following stanza to lib/dune

(subdir vendor (dirs :standard _b))

(this is equivalent to adding a file vendor/dune with contents (dirs :standard _b)).

You will also need to remove the (data_only_dirs ...) stanza from lib/dune (otherwise the (dirs ...) stanza which "lives" in vendor will be ignored).

I agree the workaround is not very satisfying and that a bit more flexibility would be useful in this case.

[grievejia]

Thanks for the tip @nojb ! It worked!

It would still be great to have a more convenient way of specifying "I want all dirs included for my foreign build", as opposed to explicitly listing out any _ directories (there could be many of them). Will leave this issue open for that but feel free to close it since it is no longer actively blocking for me.

[nojb]

It would still be great to have a more convenient way of specifying "I want all dirs included for my foreign build", as opposed to explicitly listing out any _ directories (there could be many of them).

Note that you can use globs in the (dirs ...) stanza, eg: (dirs :standard _*) will un-ignore all immediate subdirectories starting with _.

[rgrinberg]

@grievejia I'm considering this question as resolved by nojb's advice. Feel free to create more issues or re-open this one if you have any more questions.