Repository Evolution

[rowanc1]

Background

Over the past few months we have slowly started pulling repositories out of this repository into adjacent repos, this repository originally had the goal of being what has now become markdown-it-docutils and the majority of the code has now been duplicated (with the exception of state management) and improved (e.g. recursive directive parsing). That repository now has the core directives and roles that support the myst spec, and there have been other repositories that support some aspects of math parsing, and other requirements to support myst (e.g. block breaks, comments, etc.).

Repository Responsibilities

In discussion with @chrisjsewell and @fwkoch today, we are planning to change this repo into a container repository that has the following responsibilities:

• parse MyST into a AST. To start with this will be markdown-it token streams, but will be more formalized into mdast over time.
• serialize AST into a default HTML tokens
• expose an opinionated set of markdown-it plugins (e.g. that can be used in jupyterlab-myst, myst-vs-code, etc.)
• directly expose functionality to add new roles/directives that will be passed into docutils, as well as any other markdown-it plugins to add
• Handle cross-reference management through a plugin system. This will expose the ability to do cross-document reference resolution, but that is not a direct responsibility of this library.

In parallel to this @chrisjsewell and @fwkoch are looking into standardizing testing and serialization into a mdast spec, which will likely form the basis of many unit tests for "myst as a spec" that can be implemented in many different libraries/languages. Initial experimentation on this is happening in https://github.com/chrisjsewell/myst-spec and curvenote/schema#44.

Naming repositories

As a part of this, we discussed the fact that neither markdown-it-docutils nor markdown-it-myst were good names:

1. markdown-it-docutils is not actually doing the responsibility of docutils in python, which has a much wider scope. The scope of that repository is strictly parsing (using markdown-it) roles and directives. The base directives are those used in docutils/sphinx and likely common extensions to the jupyterbook ecosystem.
2. markdown-it-myst should be able to switch out the dependence on markdown-it and have no changes to the public interface (e.g. moving to https://unifiedjs.com/, which has more uptake in the MDX community etc.).

We discussed in our meeting today that we could live with (1) for now (another option thought of later is below); and (2) we should change the repository name to mystjs. myst is taken, and this nicely does not confuse it with the python implementation.

Repository Dependencies:

• markdown-it-docutils - directives and roles
• markdown-it-amsmath - \begin{equation}
• markdown-it-dollarmath - $$
• markdown-it-myst-extras - block breaks, comments, etc.
• Community markdown-it plugins (e.g. markdown-it-footnote, and others as necessary)

Notes

Note that given the name change of this repository (markdown-it-myst --> mystjs), we could move markdown-it-docutils --> markdown-it-myst, and incorporate the myst-extras pieces in there rather than having to maintain another repo/version/release etc. I personally think this makes a lot of sense, and this repository would have all the components to parse myst-specific things. It would not have things like math, footnotes, etc. which are implemented elsewhere.

@chrisjsewell and @fwkoch I am interested in your thoughts here! Personally I think this would be clearer all around and not had a tiny repo to do the extras.

Next Steps

• Closes Decide on scope and major functionality of this repository #32
• finish merge of Mystjs #35 to move to the above packages
• rename this repository and package.json to mystjs
• deprecate markdown-it-myst on npm OR rename docutils, bring in the myst-extras and rename the result to markdown-it-myst (see notes)
• Continue to add tests, standardize on mdast (likely) and get another repo together that is the myst-spec, such as it exists today.
• Figure out reference database and extension points
• Decide on role syntax!

A picture of the discussion!

[fwkoch]

I think this plan makes sense - several tightly scoped markdown-it-* plugins, brought together into a single parser/serializer in mystjs, which follows the to-be-documented myst-spec. 👍

Regarding renames:

• markdown-it-myst -> mystjs is great. To the user, this is just a library for handling MyST in js; markdown-it is an implementation detail.
• markdown-it-docutils -> markdown-it-myst - to me, this implies it's the myst plugin, which isn't the case. To meet myst spec we need all the plugins (this, amsmath, dollarmath, footnotes, etc). Maybe something like markdown-it-myst-core (and we could even bring in ...-myst-extras to that). Or we could go more explicit with like markdown-it-roles-and-directives (or both ...-roles and ...-directives 🤷).

[choldgraf]

+1 on markdown-it-myst-directives from me. I think that it's more important to tie these packages to the MyST ecosystem, rather than the docutils ecosystem. I'm also fine leaving out roles in the name even if they are bundled with directives. While they have different functionality I think semantically they're in the same group.

[rowanc1]

I am going to close out this issue, which is now better described by executablebooks/meta#838.

I think there is still some work to do to sort out the naming, but I don't think it is urgent.