Expose the includeExplanation option and/or export the tokensToHast function

[neboat]

Clear and concise description of the problem

• Exposing the includeExplanation option would enable tokens transformers to examine the explanations attached to highlighted tokens, which would facilitate building custom semantic highlighters with shiki.
• Exporting the tokensToHast function would help anyone using codeToThemedTokens to transform their themed tokens to hast.

Suggested solution

I'm using shiki to do highlighting of a custom extension to C/C++, and along the way, I implemented my own simple semantic highlighter for C/C++. I managed to port this work to the current shiki 1.0.0 beta version yesterday. But although my code works, it isn't as simple as I would like.

Ideally, I think something like the following code should work:

return highlighter.codeToHtml(code, {
    lang: lang,
    theme: theme,
    transformers: [{
        tokens(tokens) { return SemanticHighlight(tokens, theme) },
        pre(pre) { addClassToHast(pre, 'p-2.5 text-sm') }
    }]
})

Unfortunately, this code doesn't work. The main issue is that SemanticHighlight uses the explanations attached to themed tokens to perform its semantic analysis of the code. But codeToHtml seems to generate themed tokens without those explanations by default, so the semantic analysis fails. I haven't found a way change that default behavior of codeToHtml, though I might have missed something.

(Aside: I'm not sure if the tokens transformer above is correct — whether it's supposed to return a new set of themed tokens — since I couldn't find examples of such transformers.)

Instead, I currently use the following code to incorporate the custom semantic highlighting:

const tokens = highlighter.codeToThemedTokens(code, {
    lang: lang,
    theme: theme,
    includeExplanation: true
})
const _theme = highlighter.getTheme(theme)
const context: ShikiTransformerContextCommon = {
    meta: {},
    options: {
        lang: lang,
        theme: theme
    },
    codeToHast: highlighter.codeToHast,
}
const semanticTokens = SemanticHighlight(tokens, _theme)
const hastTokens = tokensToHast(semanticTokens, {
    fg: _theme.fg,
    bg: _theme.bg,
    transformers: [{
        pre(pre) { addClassToHast(pre, 'p-2.5 text-sm') }
    }]
}, context)
return hastToHtml(hastTokens)

The main issue with this code, however, is that shiki does not currently export the tokensToHast function. To make this code work, I copied the implementation of tokensToHast from shiki into my own code. That code duplication is problematic for maintenance and seems like it should be easy to avoid. I also find it strange that shiki exports codeToThemedTokens and hastToHtml but not the intermediate tokensToHast step.

If shiki either exposed the includeExplanation option through codeToHtml/codeToHast or exported tokensToHast, it would simplify and improve my code. These would seem to be simple changes to shiki that would enable some generally useful capabilities, but I don't know if there are fundamental issues with either change.

Alternative

No response

Additional context

No response

Validations

• Follow our Code of Conduct
• Read the Contributing Guide.
• Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

Contributes

• If this feature request is accepted, I am willing to submit a PR to fix this issue