fix: do not re-run jbuild syntax dune files on every run

[rgrinberg]

While this is more correct, it ends up being too slow in practice
especially since we don't have cut-off on the results

Signed-off-by: Rudi Grinberg me@rgrinberg.com

[emillon]

As I understand it, the rule that translates dune-in-ocaml-syntax to list-of-stanzas has an implicit (deps (universe)) on it (is it actually a rule in the build engine sense?), so if we want to respect it semantics we pretty much have to re-run.

Aside: I think it could make sense to switch to an explicit dependency specification for these files. For example it's common to use ocaml syntax to read an environment variable. One problem with this is that there's no good place to store that specification: it would be quirky to store that in the dune file itself; storing it in dune-project might work but we would need to store the list of dune files in ocaml syntax in the project (actually it might be a good idea in itself to declare that a project is using this escape hatch).
If we have this dependency mechanism in place, it's more correct to assume that by default, dune files in ocaml syntax have no dependencies (and don't need to get run at every iteration), but if necessary they can depend on (universe) or more precise deps.

If I understand correctly, the solution you're proposing is not to run the rule at every iteration in watch mode, but still run it every time dune build runs. This is probably an acceptable tradeoff, but it means that stopping the dune build -w process and starting it again can change the results. I'm not sure this is the case today.

[rgrinberg]

so if we want to respect it semantics we pretty much have to re-run.

Actually, we do not run universe no every iteration of the watch mode - for better or for worse.

Aside: I think it could make sense to switch to an explicit dependency specification for these files. For example it's common to use ocaml syntax to read an environment variable. One problem with this is that there's no good place to store that specification: it would be quirky to store that in the dune file itself; storing it in dune-project might work but we would need to store the list of dune files in ocaml syntax in the project (actually it might be a good idea in itself to declare that a project is using this escape hatch).
If we have this dependency mechanism in place, it's more correct to assume that by default, dune files in ocaml syntax have no dependencies (and don't need to get run at every iteration), but if necessary they can depend on (universe) or more precise deps.

Right, and there's no issue with doing that. All we need is to call the explicit build_* functions in Dune_engine and the dependencies will be tracked automatically for us.

If I understand correctly, the solution you're proposing is not to run the rule at every iteration in watch mode, but still run it every time dune build runs. This is probably an acceptable tradeoff, but it means that stopping the dune build -w process and starting it again can change the results. I'm not sure this is the case today.

Yup, this is exactly what I'm proposing. Note that this is exactly the same as universe dependencies work.

[emillon]

Suggested change

	- Do not re-run jbuild syntax files on every iteration of the watch mode. This
	- Do not re-run OCaml syntax files on every iteration of the watch mode. This

[emillon]

Internally it's known as a jbuild plugin but I think we should use the term OCaml syntax in docs.

[emillon]

Thanks for the explanation. Sounds good to me then.