RFC: Browser support & API changes

[antfu]

Currently, shiki can only work on Node.js. There is the common usage for using a syntax highlighter right in the browser for dynamic contents or building a playground. This RFC is trying to make some changes to the current implementation to make shiki able to run on both Node and browsers isomorphically. #22

I have made a hack build to replace fs.readFileSync with a sync XMLHttpRequest to fetch the data over http. You can try it at https://antfu.github.io/shiki-browser/. The main problem is that shiki will load all the grammars and themes in getHighligter which will cause high & redundant traffic & load cost in the browser.

Internal Changes

Wrap the current fs.readFileSync with an isomorphic loader like

const isBrowser = typeof window !== 'undefined'

export async function loadFile(path) {
  if (isBrowser) {
	// load from related path or cdn
  } else {
    // load from fs
  }
}

API Changes

New field themes and languages for getHighlighter to explicitly define what should be loaded. If not provided, they will work the same as we currently have.

await shiki.getHighlighter({
  // default theme
  theme: 'nord',

  // themes to load, optional
  themes: ['nord', 'github-light'],
  // languages to load, optional in Node, required in Browser
  languages: ['js', 'ts', 'html', 'vue'],
})

For users using bundlers like Webpack to build their app, they could make the loading by the bundler instead of CDN.

import shiki from 'shiki'
import nord from 'shiki-themes/data/nord.json'
import jsGrammar from 'shiki-languages/data/js.tmGrammear.json'
import htmlGrammar from 'shiki-languages/data/html.tmGrammear.json'
import customGrammar from 'path/to/custom.tmGrammer.json'

await shiki.getHighlighter({
  theme: nord,

  themes: [nordTheme],
  languages: [jsGrammar, htmlGrammar, customGrammar]
})

Also, add the ability to load grammars/themes incrementally for the highlighter

await highlighter.loadTheme('nord') // or a tm theme json
await highlighter.loadLanguage('wenyan') // or a tm grammer json

A third argument (optional) for codeToHtml to explicitly change the theme. Should raise an error if the provided theme is not loaded. This would be also useful for #33

highlighter.codeToHtml(`console.log('shiki');`, 'js', 'min-light') 

Build

Providing umd and esm builds.

Implementation

PR #109 implemented this proposal.

---

/cc @octref @orta

[orta]

I think all these changes are good ideas 👍🏻

[antfu]

About the loading in the browser, since we shipped the grammars and themes in different packages from shiki itself. Here are two solutions:

Ships within the same package

So the in the browser, we can load them with the relative path like ./themes/nord.tmGrammar.json which should work for most of the CDN providers

New option for specifying the CDN providers

Something like

await shiki.getHighlighter({
  theme: 'nord',
  themes: ['nord', 'github-light'],
  languages: ['js', 'ts', 'html', 'vue'],

  cdnProvider: 'unpkg', // jsdelivr, cdnjs, github, npm?
})

This could work in a more general way as they are loaded from absolute paths like https://unpkg.com/shiki-languages@0.2.4/data/c.tmLanguage.json. The drawback is that we would need to adapt major providers and providing a way to make custom paths.

---

Which one should we take?

[octref]

I think "Ships within the same package" is good enough. Maybe we could have a package called shiki-core that doesn't include any theme/languages, so people can load and bundle as they need.
Not sure if this is achievable by having two export targets as well.

[43081j]

Providing umd and esm builds.

This would be very nice and is the first thing I looked for when discovering this repo. You already wrote your sources as ES modules so it was surprising to see the distribution isn't also ESM.

So the in the browser, we can load them with the relative path like ./themes/nord.tmGrammar.json which should work for most of the CDN providers

I would recommend not doing this. It would be a bad idea i think to make any assumptions over where the related data files live.

Instead, I would require browser consumers to either explicitly pass the path or the already loaded object:

await highlighter.loadTheme('./my-custom/path/nord-theme.json');

As you have no idea what the production directory structure will be, and it is very likely it won't be what it was in this repo.

For example, in most of my repo's we create lazily loaded bundles which will not live at the same place they did originally on disk. So would need to specify the path to retrieve these themes and language files from explicitly.