Alignment with mystjs

[rowanc1]

I wanted to bring some visibility to a conversation @stevejpurves, @agoose77, and I had in December to discuss some possible ways to align the work here with the evolving mystjs ecosystem.

There has been a lot of work going into a javascript implementation of MyST over the last year which can now support the rich-rendering / cross-referencing that would be awesome to have directly in the editing experience in JupyterLab. There is some background about mystjs here, executablebooks/meta#838. See docs: https://js.myst.tools/ and Demo: https://myst.tools/sandbox

Currently jupyterlab-myst supports the basic markdown parsing (i.e. markdown-it plugins), but has no information about cross-cell/notebook references and limited features for reconciling citations/references/frontmatter/enumeration etc that are possible in both sphinx and mystjs.

@stevejpurves and I had a brief conversation with @agoose77 to discuss some possible ways forward to adopt more of the work that we have been putting into mystjs (parsing, enumeration, and rendering). At a Jupyter community event in Dec, @stevejpurves spun up a mirror repository (https://github.com/executablebooks/jupyterlab-mystjs) that is a different approach than the markdown-it centered approach taken in this repository. The thinking was that MyST has it's own extension ecosystem, and that it may not strictly be markdown-it in the future (e.g. reliance on unifiedjs). There are also a number of features that require global or even persisted state (e.g. references between cells) which are going to be myst-specific.

We have now gotten to a first demo/proof-of-concept of working with the full toolchain:

(please ignore styling, which I haven't figured out fully yet!)

This is showing some of the cross-referencing capability and callouts to other information like github code, citations, and wikipedia (see docs on external references and hover-cross-references).

Over the next month we will continue to push in the other repository to get that up to parity with the existing implementation here, and ideally (at least what we discussed with @agoose77) figure out a way to bring the two threads together in ~February - that might look like a major version bump here or some other way forward.

Super excited about the progress and showing off the mystjs work off in new ways, and hoping it can all come together soon! 🚀

cc @choldgraf @fperez, I will also give an update on this at the team meeting tomorrow:

• EBP Team Meeting - January 2023 🎉 executablebooks/meta#908

[welcome]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.

If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).

Welcome to the EBP community! 🎉

[rowanc1]

Some updates! I have done most of the pulling out and am getting things into a state to be pushed. I am really liking the experience so far, a bit of a gif below:

As a part of this I am also significantly cleaning up the UI components, which need to be packaged differently for JuptyerLab. Taking the chance to add some extra-things around creating demos and showcases so that-- in the future, people can mix and match these pieces.

[agoose77]

This is honestly stunning.

Not sure where this fits into the idea of a MyST-for-notebooks distribution, but Notebooks obviously benefit from executable content. I have two thoughts (probably both long-term).

1. It would be nice if there were a way to plug-into the document rendering to provide expression interpolation (like we currently have in jupyterlab-imarkdown 🤮). Jupyter Book renders JupyterLab MIME bundles, so that's the interface I'd expect here; it's fairly well known and used among different frontends (with varying degrees of support, of course). As you've probably observed, JupyterLab is built from a collection of extensions that interoperate. I'm thinking that expression rendering would be a separate extension that provides the necessary interface for the core MyST extension, ie. IExpressionRenderer.
2. I wonder whether it would be in-scope to support cell-metadata-driven figures. Right now, these are invisible in jupyterlab-myst because we try not to be too clever. However, maybe the cell output renderer could be modified to check for MyST metadata, and properly render the figure. That said I'm fairly certain that would be non-trivial amounts of work.

[chrisjsewell]

Looking cool @rowanc1 👍
definitely +1 that markdown-it (or whatever Markdown parser) should be "hid" as an implementation detail

[stevejpurves]

@agoose77

"cell-metadata-driven figures"

I'm not sure what these are exactly... is this cell metadata defining a figure label & caption, that gets associated with the [code] cell output and rendered "around" it? or is that something else?

[agoose77]

One can modify the cell metadata with a mystnb key, e.g.

"mystnb": {
 "figure": {
  "caption": "A 3D model of the detector, with ....",
  "name": "detector-3d"
 },
 "image": {
  "align": "center",
  "alt": "A time-projection-chamber detector with a MicroMeGAS anode and array of silicon detectors surrounding the exterior of the active volume.",
  "width": "512px"
 }
}

such that the code-cell outputs are embedded in a figure. It's a Jupyter Book feature, but I think some of these are in-scope for this project (even though I never got around to them!)

In jupyterlab-myst, this would require modifying the output renderer to perform this decoration

[choldgraf]

I think we should definitely start to leverage cell (and notebook) metadata to control behavior.

My main concern is that the UI/UX in JupyterLab around this is really bad. I much prefer writing YAML configuration via myst-nb text notebooks, but that requires too many extra steps w/ jupytext etc. I think that if we want to move in this direction, we should:

• State this as a strategic goal for development
• Note the challenges we need to overcome (e.g., hand-editing JSON is not acceptable long-term)
• Make it minimally possible to control behavior via cell metadata, so people have an incentive to do this
• Then focus some time on improving the UX for cell metadata to make it easier for people to do something they want to do
• After that's done we can try to refine and improve cool things you can do with cell metadata.

[agoose77]

Let me start by acknowledging that we definitely need to avoid feature creep; read my discussions as high-level ambitions rather than first deliverables.

@choldgraf are you attending the notebook format workshop? I mentioned to @rowanc1 and @chrisjsewell that I was hoping (but unsure) to attend, but my calendar is incompatible with the dates 😢. I think the importance of cell metadata relates to the notebook format, so it might be worth keeping in mind for those who are attending?

Notebook metadata editing is notoriously bad. JupyterLab has added a JSONSchema-derived UX for settings management, I wonder whether the next logical extension is for JupyterLab extensions to be able to declare a metadata schema (and JupyterLab provides the UX for free? A short-term improvement would be e.g. a front-end extension that adds a YAML view over the cell metadata. To make it slightly less painful ! I'll look into this topic. It's a broader problem than MyST.

[chrisjsewell]

I'm not sure what these are exactly.

https://myst-nb.readthedocs.io/en/latest/configuration.html#cell-level-configuration
https://myst-nb.readthedocs.io/en/latest/render/format_code_cells.html

[choldgraf]

Notebook metadata editing is notoriously bad. JupyterLab has added a JSONSchema-derived UX for settings management, I wonder whether the next logical extension is for JupyterLab extensions to be able to declare a metadata schema (and JupyterLab provides the UX for free? A short-term improvement would be e.g. a front-end extension that adds a YAML view over the cell metadata. To make it slightly less painful ! I'll look into this topic. It's a broader problem than MyST.

I think "a quick way to see and edit cell metadata as YAML" is definitely low-hanging fruit to try. I've wanted this for years haha (as much as YAML annoys me...)

are you attending the notebook format workshop

probably not - it coincides with the expected birth date of my second child :-D .

I would strongly urge you to request that the workshop discussions be available in an asynchronous fashion so that you can participate. I think that you (and others in the EB community) have a unique perspective on this because I suspect the tools in this community are making more use of cell-level data than most other tools that exist.

[stevejpurves]

I am attending that workshop @agoose77 (and would bee very happy to prep with you to get all points across/represented) - @fcollonval is heading up the organisation I believe? and I think there is a virtual component/option??

on mystnb thanks for explaining, I wasn't aware.

One point that came up in earlier conversations was should figure support be integrated further down the stack, i.e. should there be an IPython.display.Figure that pulls the label, caption and figureness down into the output rather than it being a side effect in cell metadata?

and 👍🏼 the Jupyterlb metadata editing experience -- although that is something can be maybe improved in the lab or an extension created to improve that? (ops as people have said, YAML view would be one option -- some dedicated key/value editing UI perhaps a step further?)

[agoose77]

and would bee very happy to prep with you to get all points across/represented)

I will take you up on that!

i.e. should there be an IPython.display.Figure that pulls the label, caption and figureness down into the output rather than it being a side effect in cell metadata?

Yes, I wondered about this too. This should be fairly easy; we can use the same schema subset of the MyST format. That said, I don't think these are mutually exclusive ­— having the figure metadata defined "statically" (i.e., not in code that must be executed) is probably a useful thing, and would afford a nice UX.

although that is something can be maybe improved in the lab or an extension created to improve that?

Yes, this is 100% out of scope for jupyterlab-myst: we can author this separately, or upstream into JupyterLab.

[chrisjsewell]

One point that came up in earlier conversations was should figure support be integrated further down the stack, i.e. should there be an IPython.display.Figure that pulls the label, caption and figureness down into the output rather than it being a side effect in cell metadata?

that would certainly be nice, but this is obviously only a solution for Python kernels, ideally "Jupyter stuff" should be kernel agnostic

[chrisjsewell]

I think "a quick way to see and edit cell metadata as YAML" is definitely low-hanging fruit to try. I've wanted this for years haha (as much as YAML annoys me...)

or TOML, ... lol, but yeh it feels like this should be pretty simple to implement in Jupyter, as React component or whatever

[agoose77]

that would certainly be nice, but this is obviously only a solution for Python kernels, ideally "Jupyter stuff" should be kernel agnostic

Oh, I refer to using the display-data mechanism, which is kernel agnostic. I know that xeus-cling actually defines a display API that users can invoke, but I am not sure how widespread this is in other kernels; some more primitive implementations might only cover primitive repl.

[agoose77]

Lol, talk about poor Google-fu ... jupyterlab/jupyterlab#13056

[stevejpurves]

ok -- so that's maybe in the 3.6rc0 that was released a few days ago!

[rowanc1]

First live prototype is here:

https://mybinder.org/v2/gh/executablebooks/jupyterlab-mystjs/myst-theme?urlpath=lab

If you go to examples folder, and then open the notebook in there can reproduce the screenshot above!

[choldgraf]

@rowanc1 I don't believe the branch is published, you could double check the link?

[agoose77]

https://mybinder.org/v2/gh/executablebooks/jupyterlab-mystjs/main?urlpath=lab :)

[choldgraf]

It works! Wow that is so damn cool.

[rowanc1]

From the top of the issue, we are surprisingly right on track! 🚀

figure out a way to bring the two threads together in ~February - that might look like a major version bump here or some other way forward.

@agoose77 and I just got off a call and we are planning to bring the two repositories together today and release a major version tomorrow. This allows us to have this in place before the announcement post and clean up that paragraph talking about the two repositories. Our thinking is that this will clean up any user confusion, and put all our focus on improving the next version.

Tasks following through on this that I will complete over the next hour or so:

• open an issue for sphinx-math support (splitting equations on new lines) Sphinx Math Support mystmd#208
• push the branch v1 to merge git histories
• push all tags (none actually conflict surprisingly!)
• branch current main to v0
• protect v0 in github
• switch main branch to v0 temporarily
• link to it in both readmes (v0, v1)
  • 📖 Update readme with forward pointer #76
  • 📖 Update documentation, point to v0 #59
• update all links in readme, including uninstall/disable etc.
• delete main branch, push v1 as main
• transfer over issues from other repo
• leave a message in the readme on other repo 🛑 Deprecate repository executablebooks/jupyterlab-mystjs#38
• archive other repo
• archive
  • npm
  • pypi Deprecate jupyterlab_mystjs on pypi #78 (@stevejpurves can you help?!)
• switch v0 --> main as default in github
• Fix all references on mystjs --> myst in code 📦 Rename package -mystjs --> -myst #77
• update blog post Blog post: Announcing MyST-JS executablebooks/meta#936 (review)

Tomorrow:

• test
• work with @agoose77 on any outstanding pieces
• release major version under jupyterlab-myst

Monday:

• Blog post: Announcing MyST-JS executablebooks/meta#936 goes live

[fperez]

Congrats and a huge amount of thanks to @rowanc1 @agoose77 @stevejpurves & team for all this!! I'm so excited about this, and will be providing a ton of feedback as I continue using it in production with my students this term.

Beautiful job, excited and grateful 🎉 ❤️

[rowanc1]

Thanks @fperez, I am stoked that this has merged with the original project! A few things to get back to in the next few weeks including another pass at resurrecting inline execution (#72). Had a really good chat with @agoose77 about that today!

To our early users/beta testers @mcauthorn @nthiery @fperez @stevejpurves @fwkoch @mmcky @choldgraf @echarles, thanks so much for your help so far in the jupyterlab-mystjs repository! We would love your continued help as we push on this implementation in it's new home jupyterlab-myst (no js at the end!). 🎉

We have deprecated and archived jupyterlab-mystjs, but not yet released the next major version -- that will happen hopefully tomorrow and then a blog post by @choldgraf going out on Monday mentioning this and many other new things in the myst world!

We would appreciate help in spreading the word, it is also international Love Data Week, so all very appropriate. ❤️ 🚀

[mcauthorn]

Congrats on the rapid progress! I'm happy to take a look at the new extension, as I see a ton of potential here.

[agoose77]

@rowanc1 LGTM! I think we're ready for release.

[rowanc1]

Hi all -- this has been released as v1.0.0 of jupyterlab-myst -- we announced it on a twitter and mastodon would love help boosting the visibility of this extension over the coming months! Lots of plans to make it even better, so please keep the bug reports rolling in.

I am going to close this issue now that the merge is complete. 🚀

[fperez]

🎉 huge congrats and ❤️ to everyone who got us here!

I'm super excited about where this goes next - I see enormous opportunity, esp. in tandem with the great new RTC machinery, that opens a path for a lot of new collaborative experiences with the Jupyter tools.

I'll do my part to continue dog-fooding, providing input, and signal-boosting this. Fantastic work, team!

[fperez]

BTW, quick note - any reason why the 1.0.0 tag isn't currently listed in the releases?

(I see it fine on PyPI, but it's odd that the tag isn't on GH. Did someone forget a git push --tags somewhere?

[rowanc1]

I have added the release here, thanks for the reminder! We will do better with release notes in the future now that we are all in the same repository.

I think that there is a blog post in our future as well @agoose77. :)