JupyterLab Extension to Launch Text Notebooks

[mahendrapaipuri]

This is a first attempt to launch text notebooks (which was a feature available in the legacy notebook (<7)) from JupyterLab and new Notebook interfaces. List of features added:

• A dedicated menu for Jupytext to launch text notebooks in different formats, pair notebooks and other metadata info of the extension
• All text notebook options are automatically added to JupyterLab launcher so that users can launch text notebooks directly from launcher
• Command palette have commands to launch text notebooks
• Galata UI tests have been added to test the extension in CI
• Added few more test cases to contentsManager.new_untitled to make sure we cover cases with and without jupytext.

So, users have multiple options now on how they want to launch text notebooks. A screenshot of new extension looks as follows in JupyterLab:

Note that this extension is tested and being tested in CI for only JupyterLab 4 and Notebook 7

Integration tests:

• The extension tests serve effectively as integration tests to make sure that extension and backend jupytext are working well
• README has been added to ui-tests folder on how to run galata tests
• UI tests in action here.

@mwouts @parmentelat Could you please test this patch? I have done some basic tests but I am not a user of jupytext, so not sure if everything is working well. If you think we can name the launcher icons better, please let me know.

@mwouts I tried to re-organize the source code of extension to be more clean and easy to follow. I didnt get some parts of the code, they seem to be bit vestigial but I did not remove them. If you have some time, cleaning it up and adding some comments would help the maintainability of the extension source code.

Hoping for CI to go green

Closes #1095

[parmentelat]

Hey @mahendrapaipuri

wow, you've been busy, very impressed, that sounds like a very nice piece of work !
Of course I'll make sure to give this a test; however I have to warn that now is not exactly the right timing, and it's probably going to be some time before I can get around to it, unfortunately..

so just clarifying; this no longer supports jlab3, right ? imho that's the right approach
but I guess we'll need to make sure there still is a path for jlab3 users somehow

btw thanks for pointing to playwright's UI mode, that somehow I had managed to miss at the time ;)

[mahendrapaipuri]

Cheers @parmentelat! Yes, please test it when you can.

We have moved to JupyterLab 4 and if users need to use it on JupyterLab 3 they need to pin to the last version of jupytext that is based on JupyterLab 3. This is the strategy most of the Jupyter ecosystem using.

If we would like to have these features on JupyterLab 3 as well, we can backport it to the last stable release of jupytext that supports JupyterLab 3. AFAIK there has not been a lot of change in code base between 3.6.x and 4.x in the features that we are adding in this PR. So, it should not be a too much of a hassle.

@mwouts Do you think we can do a backport and make a release for JupyterLab 3 supported jupytext?

[mwouts]

Hi @mahendrapaipuri , @parmentelat , well you've done a great job @mahendrapaipuri here again!! Thank you so much for this. I plan to test this tonight and/or tomorrow evening.

Re the backport, we could consider that, however I perceive the upgrade from Jupyter 3 to Jupyter 4 as fairly straightforward so I'd rather not do a backport unless a few users ask specifically for it.

[mahendrapaipuri]

Cheers @mwouts !!

Re the backport, we could consider that, however I perceive the upgrade from Jupyter 3 to Jupyter 4 as fairly straightforward so I'd rather not do a backport unless a few users ask specifically for it.

I agree that the upgrade from JLab 3 to 4 is pretty straightforward. But I guess a lot of users (like us) are blocked due to the unavailability of extensions for JLab 4. But what you are proposing is quite reasonable. Lets try to cut a release for JLab 4 and if there are requests from users for its support on JLab 3, we can think about backporting.

[mwouts]

@mahendrapaipuri , how do you recommend to install the extension?

I've used this:

cd jupytext  # the repo
python -m build
pip install dist/jupytext-1.16.0.dev0-py3-none-any.whl --force-reinstall

but then I see an unexpected error message in the Jupyter logs:

[W 2023-11-07 22:09:20.521 LabApp] jupyterlab-jupytext is not a valid extension
[W 2023-11-07 22:09:20.785 LabApp] jupyterlab-jupytext is not a valid extension

and also when I run jupyter labextension list

jupyterlab-jupytext is not a valid extension
JupyterLab v4.0.7
/home/marc/miniconda3/envs/jupytext-dev/share/jupyter/labextensions
        jupyterlab-rise v0.41.0 enabled OK (python, jupyterlab_rise)
        jupyterlab-jupytext v1.4.0 enabled OK

I am afraid this might be very specific to my local installation (probably at some point I installed the npm extension and now it conflicts with the py one). Do you have recommendations on how to cleanup the previously installed extensions? Anyway I'll be back to this tomorrow evening (with a brand new env!)

[mahendrapaipuri]

@mwouts That seems to me some issue with your environment. I always do a developmental install when changing source using pip install -e '.[dev]'. I recommend you to start with a clean environment.

In any case, if you get issues with extension, you can remove them from sys.prefix/share/jupyter/labextensions folder. In the present case it would be folder jupyterlab-jupytext that contains transpiled JS assets. If you want to build extension ensure that you remove existing folder at jupyterlab/jupyterlab-jupytext/labextesion so that hatch will build the extension again. Did you check out the patch branch in an existing repo? If so, that might explain the errors as well.

• Start with a clean Python environment
• Clone patch into an empty folder
• Do developmental install pip install -e '.[dev]'
• Start jupyter-lab.

Those steps should get you up and running!

[mwouts]

Thank you @mahendrapaipuri , I've been able to give a first quick try at the extension, it's really great!

Your previous instructions were very helpful. I have updated accordingly docs/developing.md in #1156 . I also found that an error that I encountered frequently in the past: AttributeError: module 'hatchling.build' has no attribute 'prepare_metadata_for_build_editable' was due to not having a recent version of nodejs (so I added that requirement to environment.yml in that PR).

Re the new Jupytext menu, I find it super useful! It's really great and so far it works very well.

I am wondering if you see some way to incite the users to select the 'percent script' over the other types (that's the script format that I would recommend now). Maybe we could have that entry at the very top of the menu, and then have a menu separator before 'Light script' and the other ones?

In the launcher I would say that there are two many Jupytext entries 😄 . Given that the 'open with' submenu introduces the concept of "Jupytext Notebook", do you think we could have a "Jupytext Notebook" section between "Notebook" and "Console" that would contain the different types of text notebooks?

A few more things that I noticed:

1. Many commands (e.g. jupytext --help, or jupytext-config list-default-viewer) print 0.7.0 twice, I imagine this is not coming from your PR, I will look into that
2. A single click on .md files opened them as notebooks directly. I was surprised because I was testing your PR on a computer on which I think I never used jupytext-config (and, jupytext-config list-default-viewer shows no override). What happens on your end when you click on a text notebook? cc @parmentelat
3. I am thinking of updating the extension version to 1.4.1 in jupyterlab/packages/jupyterlab-jupytext-extension/package.json (and add a link to this PR at jupyterlab/packages/jupyterlab-jupytext-extension/CHANGELOG.md).

I will do more testing in the coming days, and also try to make progress on the test reorganisation #1155 so that we can finally head to a new release soon!

[mahendrapaipuri]

I am wondering if you see some way to incite the users to select the 'percent script' over the other types (that's the script format that I would recommend now). Maybe we could have that entry at the very top of the menu, and then have a menu separator before 'Light script' and the other ones?

Yes, that sounds. We can have "percent script" on the top and add a separator after.

In the launcher I would say that there are two many Jupytext entries 😄 . Given that the 'open with' submenu introduces the concept of "Jupytext Notebook", do you think we could have a "Jupytext Notebook" section between "Notebook" and "Console" that would contain the different types of text notebooks?

It is possible but the JupyterLab launcher is not very customizable. If we move them to a separate category Jupytext, it will appear always at the bottom of the launcher and we lose the Python icons and it will look as below:

An alternative here would be to give the users the ability to configure which Jupytext notebooks that they would like to have in the launcher via Advanced Settings. This way we can still use Notebooks category but let users to have their preferred text notebook icons in the launcher

A single click on .md files opened them as notebooks directly. I was surprised because I was testing your PR on a computer on which I think I never used jupytext-config (and, jupytext-config list-default-viewer shows no override). What happens on your end when you click on a text notebook?

I guess you mean double clicking? For me, when I double click a "Text Notebook", it opens always as a native file (either as .py or .md). Only when I right click and use Open as Notebook, I can get the notebook interface of text notebook. And I do not have any jupytext configuration on my local env.

[mwouts]

Hi @mahendrapaipuri , ok great if you can move "percent script" at the very top of the Jupytext Menu, thanks!

Personally I do like the Jupytext section in your screenshot above. No problem if it appears at the bottom. Also I think that we can expose less text notebooks, e.g. remove nomarker and hydrogen. I would recommend to use shorter names (without Jupytext since it's on the header), like e.g.

• Percent script
• Light script
• MyST Markdown
• Markdown
• R Markdown
• Quarto

No worries if the icons don't have a Python icon. Actually that's an interesting point - these notebooks can have different languages/extensions, i.e. a percent script can well be a .R, .jl or a .sh script etc (obviously that's useful only if the kernel is for the corresponding language). From my tests I would say that currently both the Jupytext menu and the launcher entries always create a .py notebook when one selects 'script'. I know it's a bit tricky to get the appropriate extension since the kernel is chosen at the next step, so I am not sure how to make it possible to create scripts for every installed language. I think that in the old Jupytext menu for Jupyter Notebook I was using the language from the active notebook, if any. We can as well go with .py for now, since that's certainly the most common use case.

I guess you mean double clicking? For me, when I double click a "Text Notebook", it opens always as a native file (either as .py or .md).

Oh great that it works as intended on your end. I must have a residual old config here then. Thanks for confirming.

[mahendrapaipuri]

No worries if the icons don't have a Python icon.

Actually we can use jupytext logo as icons in launcher if that is something you would like.

From my tests I would say that currently both the Jupytext menu and the launcher entries always create a .py notebook when one selects 'script'.

Yes, this is because we are replacing auto with py extension here.

I know it's a bit tricky to get the appropriate extension since the kernel is chosen at the next step, so I am not sure how to make it possible to create scripts for every installed language.

Exactly!! I think this is something we should try to improve for the next iteration. Created #1157 to keep track of this issue.

[parmentelat]

Hiya folks

I still have not been able to test @mahendrapaipuri's super appealing work, but have some thoughts/comments from reading this thread so far

• I like the suggestion

  An alternative here would be to give the users the ability to configure which Jupytext notebooks that they would like to have in the launcher via Advanced Settings. This way we can still use Notebooks category but let users to have their preferred text notebook icons in the launcher
  indeed in my case I would gladly pick md:myst and py:percent - or auto:percent- as the 2 entries that I'd like to find in the Notebook category of the launcher
  I understand this may be additional work, and I can open a separate issue on this if there's a consensus; or it may be made part of Create language specific Text notebooks #1157, if we think of it as a second pass on this first iteration of the extension revamping

• secondly, unrelated but timely, I'd like to discuss the ability to download the current notebook in ipynb format
  in my usual setup we expose notebooks in, say md:myst, that are NOT paired; in the classic notebook we had the ability to download the ipynb, and that is sometimes used by the teachers; somehow this has been lost in translating to jlab4; please let me know what you think about that one, probably best in another issue, but I wanted to take the opportunity of this major overhaul to raise the question

in any case, thanks again so much @mahendrapaipuri for this very welcome contribution !

[mahendrapaipuri]

@mwouts @parmentelat I have added ability to configure both Category and formats of text notebooks that will appear in the launcher along with some other changes that we agreed upon until now.

I have also added a small docs about extension with few screenshots. Please test the patch when you can and let me know.

[mwouts]

Awesome thank you @mahendrapaipuri! I will definitely test this over the weekend. Looking forward to merging this!

[codecov]

Codecov Report

Merging #1154 (3612e89) into main (b6f0ae6) will increase coverage by 0.11%.
Report is 1 commits behind head on main.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #1154      +/-   ##
==========================================
+ Coverage   94.99%   95.11%   +0.11%     
==========================================
  Files          29       29              
  Lines        4419     4419              
==========================================
+ Hits         4198     4203       +5     
+ Misses        221      216       -5     

see 2 files with indirect coverage changes

📣 Codecov offers a browser extension for seamless coverage viewing on GitHub. Try it in Chrome or Firefox today!

[mwouts]

This is very helpful! Thanks!

[parmentelat]

about jlpm watch it is indeed very convenient, just keep in mind the .ts part, because it won't trigger if you only change a .json though, so in these cases it is important to kill it and restart it, at least that was my experience; I'm unsure about .css files, so out of safety I am used to relaunching jlpm watch too, it may be unnecessary.

[mwouts]

Very impressive! And helpful as it will allow us to make future changes! Thanks @mahendrapaipuri