queries/nix: add injection rule for python test scripts, fix rules

[pacien]

No description provided.

[pacien]

This adds a missing injection rule for the NixOS VM test scripts written in
Python.

However, I could not get this rule to apply without commenting the one that
preceeds it.

Re-ordering the rules in the file does not seem to change anything either.

Is there a way to set some priority on the rules?

This would also be useful to allow the comment rule to take precedence, so that
syntax highlighting can be manually overridden.

(CC past file committers: @nrdxp, @the-mikedavis, @farwyler)

[pacien]

Here's an example of the issue I'm observing due to the lack of
priority/ordering of the rules: some snippets seem to have some incorrect
syntax highlighting.

{ pkgs, ... }:

{
  # highlighted as python as it should ("writePython3Bin" rule)
  package = pkgs.writers.writePython3Bin "test" { } ''
    from sys import stdin
    print(i**2 for i in range(1, 10))
  '';

  # highlighted as python as it should (comment rule)
  fromComment = /* python */ ''
    from sys import stdin
    print(i**2 for i in range(1, 10))
  '';

  # highlighted as bash as it should ("*Script" rule)
  some.dummyScript = ''
    from sys import stdin
    print(i**2 for i in range(1, 10))
  '';

  # highlighted weirdly? Should be highlighted as python (comment rule)
  another.dummyScript = /* python */ ''
    from sys import stdin
    print(i**2 for i in range(1, 10))
  '';

  # highlighted weirdly? Should be highlighted as python ("testScript" rule)
  some.testScript = ''
    from sys import stdin
    print(i**2 for i in range(1, 10))
  '';

  # highleighted weirdly? Should be highligted as python (comment rule)
  another.testScript = /* python */ ''
    from sys import stdin
    print(i**2 for i in range(1, 10))
  '';
}

[nrdxp]

I think I saw something like this in my own queries too. IIRC this might have something to do with the injections.combined, which does a little bit more than I want it to. perhaps @the-mikedavis has some insight?

[archseer]

I think all of these snippets should drop injections.combined it's intended for filetypes like erb or handlebars where all the fragments in the file combine into one large document, whereas here every individual case should be parsed separately

[pacien]

I think all of these snippets should drop injections.combined
it's intended for filetypes like erb or handlebars where all the
fragments in the file combine into one large document, whereas here
every individual case should be parsed separately

With injections.combined removed, the last three examples do not seem to be
affected by the Python rule at all anymore.

What's the correct way to set overrides / priorities without combining?

[the-mikedavis]

Stanzas that are higher in the file will have higher precedence but we will also need to tune the comment injection rule on L5-7. The dummyScript with a comment case is failing because the stanza on L11 starts matching earlier than the comment stanza (L5). We can add another stanza right next to the comment stanza that starts capturing at the same time, so it will have higher precedence:

; mark arbitary languages with a comment
((((comment) @injection.language) .
  (indented_string_expression (string_fragment) @injection.content)))
(binding
  (comment) @injection.language
  expression: (indented_string_expression (string_fragment) @injection.content))

[pacien]

Thanks, I've applied the suggestions;
Everything is now working as expected!

[nrdxp]

fwiw, without injections.combined highlighting stops at the first instance of string iterpolation, which is why I added it in.

[pacien]

fwiw, without injections.combined highlighting stops at the first
instance of string iterpolation, which is why I added it in.

This still seems to be working without:

[pascalkuthe]

interpolation like this is difficult since tis not really valid python and instead essentially an injection insice injection.

I guess the nixos grammar does actually endup producing multiple syntax nodes here so you would essentially want combining injections within a single TS node (but not globally). That doesn't currently exist in helix however.

I think using combined injection does actually represent that more correctly for now. The issues you showed seem odd I don't have time to investigate right now but that looks like an actual highlighting bug. This seems similar to some of the issues I saw (and fixed) in #7621. Jjust to be sure:

Are you using the latest master build?
Otherwire if you could upload a minimal reporduciable example file (with some more details) that would be helpful so I can look into whats causing that.

[pacien]

I just tested again rebased on master and without removing the combining,
applying @the-mikedavis' fixes for the order and start of the matching.

It seems to work as expected with the example above in this thread!

[the-mikedavis]

Having scopes for combined injections so that they're limited to some part of the subtree that you select with queries could be useful for other languages (in particular HEEx). That's a feature that we might want to suggest upstream first though since it would benefit any tree-sitter consumer using injections

[pascalkuthe]

I think we would only need to agree on the query syntax tough. The implementation should be entirely possible on our end.