Thoughts on fixing submodules

[lf-]

It's very difficult to depend on tree-painter except by path, since cargo submodule support is awful, and it takes multiple minutes to full-depth clone everything on an 800mbit connection: rust-lang/cargo#7987 rust-lang/cargo#1171. This is a huge usability issue, unfortunately. Also, the rust bindings of any given tree-sitter parser are not necessarily maintained or consistent.

I think that Helix has fairly exhaustively explored the other ways to achieve it: it does all three of the following:

• release tarballs which contain the tree-sitter grammars. Those are then built by build.rs https://github.com/helix-editor/helix/blob/master/.github/workflows/release.yml#L36-L48

  These deal with OS packaging use cases, and probably would also be useful for some other things. You can see the release tarballs in action in the nixpkgs packaging for Helix (operating system packaging rather than development packaging): https://github.com/nixos/nixpkgs/blob/2905eb5343e5d86fa281c58b1344e2195fd583cb/pkgs/applications/editors/helix/default.nix#L7-L13

• providing the grammars in a runtime directory with an environment variable. This means that an external build system can provide them, which is done by the fancier nix packaging in the Helix repo: https://github.com/helix-editor/helix/blob/master/flake.nix#L121-L143

  These are built by nix as .so files: https://github.com/helix-editor/helix/blob/master/grammars.nix

• Using build.rs to clone and build the parsers: https://github.com/helix-editor/helix/blob/master/flake.nix#L72 https://github.com/helix-editor/helix/blob/master/helix-term/build.rs#L7-L30 https://github.com/helix-editor/helix/blob/master/helix-loader/src/grammar.rs#L251-L283

In sum, it looks like this will require some build engineering work, but it seems necessary to build it with Nix or really, depend on it in any scenario.

[matze]

For wastebin, I added tree-painter as yet another submodule. It's not nice at all from a setup point of view but it avoids the horrendous clone times.

P.S.: I saw your concerns regarding performance. Not sure what's wrong but when comparing syntect with tree-sitter, highlighting becomes an order of magnitude faster in release mode. Highlighting option.rs takes ~160ms with syntect but merely 13ms with tree-sitter.

[kaj]

This does not seem to be a problem when depending on tree-painter through a crates.io release (currently 0.0.0), but it does seem to be a problem when trying to get newer code by git dependency?

# No problem
tree-painter = "0.0.0"

# Problem?
tree-painter = { git = "https://github.com/matze/tree-painter" }

Is that so? Or is it just that there were fewer modules in the 0.0.0 release?

[matze]

Or is it just that there were fewer modules in the 0.0.0 release?

Kind of. The thing is, if I want to make a crates.io release all dependencies must be on crates.io as well and of course have matching tree-sitter version requirements. Unfortunate reality is however that most grammars are not very keen on regular updates and some do not even provide Rust "bindings" even though they could. Hence the submodules which I could update on my own but then prevent me from making proper crates.io releases.

[kaj]

Hm. Maybe there could be a tree-sitter-parser-collection crate that does all the required magic, that various crate that uses the parsers (like this one) could depend on?

[lf-]

that's something I'm working on slowly with lf-/gridlock, actually.

[matze]

If that's a repo, it is inaccessible :-(

[lf-]

@matze apologies for that, it is an unfinished side project and I was super busy and didn't get around to finishing it enough to make it usable-ish. It's now public: https://github.com/lf-/gridlock