Soft illegals

[brunophilipe]

Instead of ignoring illegals completely, using "soft illegals" will cause highlight.js to use the illegal expressions as an alternate end expression. This way, it will still refuse to highlight illegal expressions, but without bailing out completely.

Please note my experience with JavaScript is limited. Perhaps there is a better way to achieve this behavior. However I tested it and it works very well! This is interesting for my use case because I want to be able to highlight small chunks of code at a time in my code editor, which are not necessarily correct, as the user might still be composing them.

---

For example. This illegal multi-line string is currently highlighted when using "ignore_illegals":

char *string = "foo\
bash
baz";

---

Using soft illegals, the string expression is highlighted up to the "bash" word, after which the expression ends. The resulting highlight is better than not having any highlight at all, and actually matches exactly what Xcode highlights for the same expression:

In Xcode:

Cheers!

[marcoscaceres]

Interesting suggestion. Some API modifications while I think this through a bit.

[marcoscaceres]

so, instead of changing the function signature in this manner, I suggest you instead change it to accept an an object as the last argument:

function highlight(name, value, ignore_illegals, continuation, options) {
}

Then pass the options to compileLanguage:

compileLanguage(language, options);

And then check the options.soft_illegals... I would also change soft_illegals to treat_illegals_as: "soft", that way, it's more extensible in the future.

[joshgoebel]

Definitely agree.

[joshgoebel]

Also we really should have at least one test for this to show the behavior both ways... and how does this intent to interact with ignore_illegals now? (since it's a sep option)

I assume this would be paired with ignore_illegals: false, right?

• ignore_illegals: false, treat_illegals_as: "soft": illegal ends the match
• ignore_illegals: false, treat_illegals_as: "hard"/null: illegal raises

And the behavior with ignore_illegals true is that it just blows by them and keeps looking for the final end match, yes?

[joshgoebel]

Isn't this why we have the reStr helper?

[joshgoebel]

Definitely agree.

[joshgoebel]

Having two different settings does seem messy... I wonder if it wouldn't be clearer to change the parameter to "handle_illegals": ["ignore", "end_match", "error"]... we could grandfather in true and false for ignore/error...

[joshgoebel]

This needs to be held until after #2277 in which case we can just come up with the perfect clear name and then add it as a boolean flag.

[joshgoebel]

Actually this code base would have to be entirely reworked, so I'm closing this PR for now. It would likely need to interface with processLexeme (at run-time) not hack the modes at compile time. We only compile modes a single time, so the mode itself needs to be static (post-compile) and cannot be dependent upon run-time config constraints, like it is in this patch.

for my use case because I want to be able to highlight small chunks of code at a time in my code editor

Also, just FYI this is not really a supported use-case. Of course you're free to do whatever you want with the library, but I'm hesitant to add features just to support a use-case we actually don't encourage or support.

I'd suggest we first consider how to possibly solve this with a plugin rather than adding it to the core highlighter. Perhaps an on:Illegal hook/callback of some kind that plugins could tap into... OR I've also been considering mode compile callbacks... so a plugin could perhaps mess with the mode compilation and turn illegal into an end variant (like this PR does)... or course this would have the same caveats I mention above, but that might not be a concern for the users of the plugin.

If we want to continue the discussion I'd suggest opening a new issue thread and pointing to this PR for context.