Initial draft of myst-spec announcement.

See #538

docs/updates/2022-04-15-myst-spec.md

@@ -0,0 +1,81 @@
+# MyST Spec & MyST in Javascript
+
+```{post} 2022-04-15
+:author: Rowan Cockett
+```
+
+MyST (Markedly Structured Text) is designed to create publication-quality, computational documents written entirely in Markdown. The main use case driving the development and design of MyST is JupyterBook, which helps you create educational online textbooks and tutorials with Jupyter Notebooks and narrative content written in MyST. The documentation engine that is used for JupyterBook is Sphinx: a widely used tool for Python documentation based on [ReStructured Text (RST)](https://en.wikipedia.org/wiki/ReStructuredText). The fantastic thing about the Sphinx community is that there are ways to create new extensions and use these extensions _directly from the RST language_; this extensibility and community was what we aimed to harness when bringing this super-power into Markdown.
+
+MyST is a superset of [CommonMark](https://commonmark.org/) (the standard form of Markdown) and allows you to directly create “directives” and “roles” as extension points in the language. `directives` are block-level extension points, like callout panels, tabs, figures or embedded charts; and `roles` are inline extension points, for components like references, citations, or inline math.
+
+````md
+```{directive} Argument
+The *content* of a directive can have {role}`with content`
+```
+````
+
+If you want to dive deeper into the MyST syntax and features available, [see here](https://myst.tools)!
+
+There have been uses of Jupyter Book and MyST from heliophysics to music classification (see [the gallery](https://executablebooks.org/en/latest/gallery.html))! The simplified authoring environment provided by MyST and the workflow tools that stitch together Notebooks into a published website, from a few lines of configuration, have been key to the adoption and use by the community.
+
+As our community grows, we want to make sure that these foundational tools and syntax we are introducing with MyST can support the diverse usage and needs of our community. Beyond JupyterBook, some examples include software documentation, directly exporting to LaTeX and PDF, supporting new thesis or paper formats online, integrating into development environments ([like VSCode](https://github.com/executablebooks/myst-vs-code)), editing and seeing changes [directly in Jupyter](https://github.com/executablebooks/jupyterlab-myst), and editing using [visual WYSIWYG tools](https://curvenote.dev/editor). To support these tools and applications, MyST needs to evolve from being a single language implementation in Python to have multi-language, multi-implementation, multi-community support. This requires some changes and enhancements to MyST and a pathway to develop it into a standard.
+
+Today we are excited to talk about four things:
+
+1. ✨ A new **website**, <https://myst.tools>, that introduces the MyST markup language, complete with demos, docs, pointers and more. This will provide a centralized place to learn MyST and pointers to other related tools to help with your documentation, writing, and publishing.
+2. 🧪 A set of 1400 unit tests to support the **MyST Spec** in a language and implementation agnostic way.
+3. 🥇 A brand new **javascript** implementation, `mystjs` and supporting ecosystem of `unified-myst`, that passes all 1400 of those tests!
+4. 👩🔬 MyST **enhancement proposals** to improve the markup language and ecosystem over time.
+
+First, go check out [myst.tools](https://myst.tools) and please give us some feedback on Twitter ([@ExecutableBooks](https://twitter.com/ExecutableBooks)) or through [GitHub](https://github.com/executablebooks/meta/discussions), and help spread the word about these new developments in MyST!
+
+# Moving towards a MyST Spec
+
+To build on the MyST markup language and to make the ecosystem as rich and interoperable as possible, we need to formalize three formats (1) the MyST syntax, to ensure MyST works as expected across languages and implementations; (2) the MyST abstract syntax tree (AST), to promote an ecosystem of transformations and exports to diverse formats; and (3) the semantic HTML output and CSS class structure, to promote web-accessibility and interoperability of themes. Additionally, we require a way to improve and enhance these formalizations over time in our multi-stakeholder community.
+
+First, the syntax that we are supporting is what exists today without any changes: this is what you have been using as you move from RST to MyST, or document your notebooks and courses in JupyterBook. We aren’t making any changes to the MyST syntax for now, and are instead formalizing the existing markup. We are doing this by introducing a list of tests to cover the MyST Spec, you can read more about that at <https://spec.myst.tools>. These tests cover three target formats at the moment: `myst`, `html` and `mdast`. An example test case for a header looks like:
+
+```python
+cases:
+  - title: CommonMark headers
+    mdast:
+      type: root
+      children:
+        - type: heading
+          depth: 1
+          children:
+            - type: text
+              value: Heading!
+    myst: |-
+      # Heading!
+    html: |-
+      <h1>Heading!</h1>
+```
+
+Markdown AST, or `mdast`, is the intermediate format that is new for MyST, and builds upon the existing [mdast spec](https://github.com/syntax-tree/mdast), which is supported throughout the [unifiedjs](https://unifiedjs.com/) Javascript community with hundreds of [existing transformations](https://unifiedjs.com/explore/), [utilities](https://unifiedjs.com/explore/keyword/unist-util/) and [serializers](https://unifiedjs.com/explore/keyword/rehype/). `mdast` is simple, with a `type` defining the node optional properties on the node and optional `children` (a leaf node has a `value`). Beyond CommonMark and GitHub Flavoured Markdown, MyST introduces new directives and roles (like admonitions, citations, math) following existing standards where they are defined. `mdast` is serializable to JSON or YAML, and can be effectively shared between projects. In fact the [spec](https://spec.myst.tools) has a [versioned JSON file](https://unpkg.com/browse/myst-spec/dist/) that is now included in both our Python and Javascript implementations today.
+
+### Markdown Abstract Syntax Tree, `mdast`
+
+Standardizing on `mdast` allows all sorts of new transformations or utilities to be created to aid in analyzing the tree (e.g. how many words, admonitions, or references) producing helpful user interfaces in VSCode or Jupyter that efficiently analyze/lint your writing (e.g. [repeated words](https://github.com/retextjs/retext-repeated-words), or [inclusive language](https://github.com/retextjs/retext-equality)). So although this is still fairly low down the stack, we are excited about the types of plugins and extensions that this will enable for user experiences in the MyST ecosystem. Importantly, this also works across our new Javascript implementation, `mystjs`, which will be used many places (e.g. in Jupyter, VSCode).
+
+Throughout the documentation on <https://myst.tools> you will now be able to edit any MyST snippet and directly see the rendered demo, the AST, and the rendered HTML. In the first release of the MyST Spec, we have also made efforts to improve the semantic HTML (e.g. using `figure`, `figcaption` and `aside` in place of `div` with a class name). This is not only best practice, it helps screen readers, search engines, and other user devices determine the significance of any block of content. You are also, of course, free to transform the `mdast` after it is created if you require additions or custom transformations.
+
+## MyST in Javascript
+
+The newest addition to the MyST stack is a [javascript implementation](https://github.com/executablebooks/mystjs) of the MyST Spec called `mystjs`, which is an opinionated wrapper on [unified-myst](https://github.com/executablebooks/unified-myst). There are many tools, including VSCode and Jupyter where it is necessary or more practical to use Javascript to parse and render MyST. There is also a vast community of existing tools that build on some of the concepts that we have introduced above: notably the [unifiedjs](https://unifiedjs.com/) Javascript community which provides standard ways to **parse → transform → serialize** documents through plugins. The components of the parser are accessible to interrogation through [unified-myst](https://github.com/executablebooks/unified-myst) so that you can put together your own flavour of MyST parsers, and there is a robust extension architecture already in place that is heavily inspired by the Sphinx ecosystem.
+
+These plugins are all built around transformations and translations between ASTs (e.g. Mardown is [mdast](https://github.com/syntax-tree/mdast), HTML is [hast](https://github.com/syntax-tree/hast)). `mystjs` and the `unified-myst` ecosystem of packages are already used in the VSCode plugin and the alpha version of our Jupyter extension that renders MyST directly in your Jupyter Notebooks. Take a look at the documentation, <https://javascript.myst.tools>, and if you are a javascript user test it out on some content. Some of our next steps here are to join this up with existing tools in the javascript ecosystem to support more export formats for MyST to HTML, DOCX and {math}`\LaTeX`. Follow along for updates on Twitter [@ExecutableBooks](https://twitter.com/ExecutableBooks).
+
+## MyST Enhancement Proposals
+
+We would like to see the MyST Spec, language and themes improve over time, with more support for new export types (e.g. Microsoft Word, LaTeX and connections to pandoc), compatible themes (e.g. [book theme](https://github.com/executablebooks/sphinx-book-theme), [sphinx design](https://github.com/executablebooks/sphinx-design), [furo](https://github.com/pradyunsg/furo)), and new ways to edit MyST in your existing IDE or with WYSIWYG tools (e.g. in [VSCode](https://marketplace.visualstudio.com/items?itemName=ExecutableBookProject.myst-highlight) or [Curvenote](https://github.com/curvenote/editor)). To support this process we have created a MyST Enhancement Proposal (MEP), which will govern additions or evolutions to the MyST syntax or `mdast`, as well as the directives and roles that are core to MyST (e.g. `admonition` and `ref` are core, but `prf:theorem` isn’t yet!). The first few proposals are already up for consideration, including simplifying admonition titles and a proposal for inline arguments for roles. As a component of each MEP, we will include upgrade paths for your content and a deprecation cycle that makes this process automated.
+
+## Looking into the future
+
+We created MyST to provide an approachable content authoring language that is extensible, pluggable, and has a direct focus on interactive & computational content. We are stunned by the quality and breadth of the interactive, computational teaching resources that educators are creating, and proud that these tools are getting use beyond our initial targets. We want to help support a world where first-class interactive, computational content can get embedded in everything from a personal blog, to technical documentation, to a publishable scientific article or book.
+
+The standards and community processes that we are introducing will help support this ecosystem and ensure that we have input from a wider range of stakeholders, while making actionable improvements to the tooling and specifications around MyST.
+
+Thanks for your interest and support as we work on MyST, if you haven’t tried MyST yet, give it a try today in [our documentation](https://myst.tools), or get started with [JupyterBook](https://jupyterbook.org).
+
+You can follow our updates on [@ExecutableBooks](https://twitter.com/ExecutableBooks) on Twitter.