How to handle embedding other language syntaxes

[nk9]

We now have a PR from @deathaxe with syntax embedding for recipes. 🎉 It solves the issue in #1 by relying on the embedded shell syntax to properly handle backslash-escaped newlines. It still allows lines to have a just prefix (@ and -) by using meta_prepend with the recipe-content-modifiers context.

I would like to hear from @casey his thoughts on receipe syntax highlighting. The primary issue I see is that just allows users to change the default shell using the shell and windows-shell settings. If a user set their default shell to ruby for example, and then the syntax proceeded to apply Bash highlighting to all the recipes in the file, that would probably be annoying.

A workaround in that case could be to use a shebang (and we'd have to add Ruby embedding to our syntax). I'm just not sure how big a deal this is. It's technically not correct to highlight recipes as Bash when there's another shell set, but it's just not fixable given the constraints of regex-based syntax highlighting. (LSP might help, but that's not the syntax engine we're using.)

So the options as I see it are:

1. Use Bash by default and users who want correct syntax for other languages will just have to include a shebang. We would notice shebangs which we don't have an embedded syntax for and leave those un-highlighted.
2. Only apply syntax highlighting (other than interpolation and line prefixes) when the shebang is present.

The same issue in miniature is present in backtick strings, which (like Bash) invoke a shell and return the result. Those are handled as strings in the syntax at present, but if we choose option 1 above, they should probably be highlighted as shell too. No shebang workaround there, though: they will just always be wrong if a file defines a different shell.

[deathaxe]

The primary issue I see is that just allows users to change the default shell using the shell and windows-shell settings.

This literly rules out any kind of syntax highlighting within receipts as each syntax (Shell/Python/Ruby/...) comes along with fundamentaly different concepts and rules. It wouldn't make sence or create a good UX to try to highlight anything but "just" related stuff. I don't think even strings etc. would be useful.

ST's syntax engine can't handle such dynamic use cases.

The only possible solution would be to create Just syntax for each default shell (e.g.: Just (Ruby), Just (Python), ...). It would however require a dedicated file extension for each of them or a plugin which can somehow detect the shell value and set corresponding syntax dynamically to one of the derivates.

[nk9]

I just learned this was discussed at length last year in the vim-just repo. @casey seemed to prefer a general "recipe text" scope for all recipes, save interpolation and line prefixes. So pretty close to option 2 above. We could still apply more precise syntax to recipes which have a shebang.

I'll work on adapting the code in the PR to this approach.

[nk9]

I've adapted your PR, @deathaxe. I'm just trying to get it working, so I've left the "shell by default" behavior in place for now. And that actually works. However, the Python embedding does NOT work. I can't figure out why, since the two seem to be implemented the same way. Do you see anything wrong in the branch?

Note also that, in line with your comment above, I'm only planning to provide embedded highlighting for recipes with an explicit shebang. Any advice on what scope to use for recipe content which doesn't have a recognized shebang?

Lastly, I've done some testing and the recipe content modifiers (@ and -) only work on recipes without a shebang. So given the above, they won't be needed in the embedded prototype. It will only be interpolations that need to be prepended to the embedded syntaxes. I've removed that from the Python one already, still a TODO.

[deathaxe]

Do you see anything wrong

The comments which start at the beginning of a line after python shebang just pop python syntax as it is how the end of receipts is defined.

Changing it to

script3:
    #!/usr/bin/python3
    # <- source.python.embedded.just comment.line.number-sign.python punctuation.definition.comment.python
    #  ^^^^^^^^^^^^^^^^^ source.python.embedded.just comment.line.number-sign.python
    print("hello {{ ARGS }} world")
    #^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ source.python.embedded.just

should do the trick.

... only planning to provide embedded highlighting for recipes with an explicit shebang

This is the only safe way to go with all kinds of scripting languages being used as default dynamically.

Any advice on what scope to use ...

One possibility would be to scope body of all receipts meta.receipt or meta.function.just and just leave the unknown parts alone with that meta scope, while adding source.<syntax> for those with shebang.

It would probably also not hurt to just keep those lines unscoped. Wouldn't add a pseudo source.other scope or something like that.

I've removed that from the Python one already,

Whatever works.

[nk9]

Thank you for the advice, I'll look at a meta scope.

You're right about the embedded syntax popping on a ^\S line, good catch. But moving as suggested (and, in fact, just doing away with the syntax test comments entirely) doesn't seem to work:

The Python (for Just) file is right next to the ShellScript file, which DOES work. How should I debug this? 😬

[deathaxe]

ST's Python syntax is of version: 1 while the extending one in Syntax/Embeddings is of version: 2. Removing that line should fix it (until ST is upgrading python to version: 2 in a future release).

An inherited syntax must always be of same version as the base syntax.

[nk9]

Yes, that was it! Thank you!

[nk9]

OK, Shell (sh/zsh/bash) and Python are implemented. Thanks again for your help!