helix: unable to fetch / build custom tree-sitter grammars

[flokli]

Describe the bug

tree-sitter grammars are a bit of an odd beast. At runtime, they are very platform-specific .so (or .dylib) files, which is why the community seems to have settled on referring to source code of some javascript package, plus some c code, and compiling this every once in a while on the target machine running the editor.

Helix compiles most of its grammars during build time, but also allows custom grammars to be configured. These configs point to checkouts of said node modules.

It ships some hx --grammar fetch and hx --grammar build (sub)commands, which theoretically would (re)build existing grammars (shouldn't be necessary in our case for the ones shipped with helix itself), but also include new ones manually added to ~/.config/helix/languages.toml.

This currently fails:

Fetching 124 grammars
124 grammars failed to fetch
	Failure 1/124: Could not create grammar directory "/nix/store/yhb17fbjgsfp7mgvn5igxkz3fjayfinb-helix-22.12/lib/runtime/grammars/sources/awk"
	Failure 2/124: Could not create grammar directory "/nix/store/yhb17fbjgsfp7mgvn5igxkz3fjayfinb-helix-22.12/lib/runtime/grammars/sources/rust"

I'm not sure it's because we remove the grammars initially, or because the "upstream grammars" Helix itself is bundled with should be skipped (they can't change anyways, because their git rev is defined in the languages.toml of the source code we built helix with).
Maybe helix should only fetch/rebuild new ones in a local directory?

Another orthogonal thing would be to be able to point helix to a .so/.dylib file directly (even though it's platform-specific), rather than this compiling at runtime - but that'd be a feature request for helix, and I'm not sure it's a good idea.

Steps To Reproduce

Steps to reproduce the behavior:

1. Add a new grammar section to your ~/.config/helix/languages.toml, as described in https://docs.helix-editor.com/guides/adding_languages.html#adding-languages
2. Run hx --grammar fetch
3. Observe the build errors

Expected behavior

I'd expect helix to fetch (and later compile) the added grammar.

Notify maintainers

@danth @yusdacra

Also cc @the-mikedavis

[flokli]

It looks like nvim-treesitter has a setting to configure it to compile grammars into a separate directory, maybe helix could fall back to also consider ~/.local/state/helix (XDG_STATE_HOME) instead, especially if the global path is read-only?

[the-mikedavis]

This is a Helix issue rather than Nix or nixpkgs and it was covered by helix-editor/helix#3346. It's also relevant for other package managers since they usually install the runtime directory under /opt or /etc or somewhere that needs superuser rights to modify.

It's been fixed on master: helix-editor/helix#5411

[flokli]

Thanks for the link! I'll check how a more recent version of helix will behave.

[flokli]

Ah booh, it's a bit more complicated to test, as the tree-sitter grammars are not normally part of the repo, and only in the release tarballs 🤔

We could potentially nixify all that build as well, but it'll be a lot of work. Do you plan to do a new release soon, by any chance? ;-)

[the-mikedavis]

If you have flakes enabled you can use the Helix flake to run the latest master: nix shell github:helix-editor/helix

The next release is currently planned for 2023-03-31

[flokli]

I don't have flakes enabled, but I'll be happy to check the next release, if it's just around the corner.

[flokli]

So hx --grammar fetch and hx --grammar build now seem to work. I still couldn't get that custom grammar I'm testing with to work, but there's now a bunch of .so files in ~/.config/helix/runtime/grammars - so I guess it'd work in theory, and might very well be a problem with that specific grammar.

Let's close this one.