source_wrapper overshoots non while based embdedding

[matter123]

LatexWorkshops grammar embeds c++ using a begin/end pattern. This causes the source_wrapper to not detect the end. microsoft/vscode#76603 and James-Yu/LaTeX-Workshop#1476. The markdown grammar does not suffer from this because the while check cuts off the end pattern of source_wrapper.

[jeff-hykin]

Current workaround

If the parent language (in this case LaTex) includes the source.cpp#initial_context instead of source.cpp and manually adds the source.cpp tag to it's range it should work. Later this week I might pull in LaTeX and add that.

This same solution stands for embedding the shell, perl, and dockerfile grammar. However C, objective-C, objective-C++ shouldn't have this issue.

General problem

This is tough since it is a somewhat unresolvable conflict. Source wrapper needs to wrap the whole language, but it doesn't know when then language ends. I had the same issue when I was embedding the Shell grammar inside of the Dockerfile grammar. Its what started me looking into the textmate while feature.

The while method is the only surefire way of because it takes full priority over any other pattern. Other methods are not fullproof because code fragments like class { will not close the C++ range and it will overflow.

Even assuming the embedded code has closed all the pattern ranges, one solution could be to do statement matching, where every individual statement is wrapped inside of a source.cpp. I'm hoping to get to this point with the C++ grammar but its not quite there yet.

The most ideal solution would be for the vscode-textmate to add a tool like the while pattern that takes full priority over the other patterns, and can force-close all lower-level ranges. I was hoping to work on such a feature but I have no idea how difficult it is to implement. It'll probably be a month before anything useful is accomplished.

I think you know more about the vscode-textmate parser than I do. I went ahead and added you as a collaborator on https://github.com/jeff-hykin/vscode-textmate. Feel free to push directly to master or merge branches into master. In a week or so I'll switch the cpp-textmate repo back to using the standard vscode-textmate, and add a separate command that will run the tests with the experimental version.

This kind of a feature could be really really useful for preventing overflow bugs, which would be nice for basically all macros/preprocessor commands in every language.

[jeff-hykin]

I removed the source wrapper. Instead of the issue falling to each language I think they're now the responsibility of the language that is embedding C++ (meaning this is no longer an issue for C++). At some point I'll remove the source wrapper code from the shell syntax and the textmate_tools.rb