Restore TOC code cells feature

[Rmarieta]

Problem

Edit : You can use this TOC again with JupyterLab >= 4 installing this extension : https://github.com/jupyterlab-contrib/jupyterlab-code-toc

I've been working with the table of contents feature in collaboration with teachers and assistants for some time, and I have had the feedback multiple times that the old table of contents that included the code cells and "non-heading" markdown content was preferred. By old table of contents, I mean the following table of contents that was the default toc in older version of jupyterlab such as 3.6.3 and that got refactored around July 2021 :

I have looked for issues or explanations in the commit history as to why the removal of the code cells and other extra features was decided at that time but I could not find a proper explanation. For the code cell toggle option in particular, would it be possible to restore that in the newer versions of the table of contents ? Removing the code cells from the table of contents and going for a newer version without an alternative is surprising so if there is an underlying technical issue justifying that decision, I would be happy to learn about it.

Proposed Solution

To restore that feature, since the way the toc is built has changed from using generators and monitoring changes in the notebook to connecting to changes in headings and retrieving them, the way the code cell content is retrieved would have to be adapted. I would be open to implement that feature myself since I've been building extensions with the current and the previous toc extensions, and I've already restored the old toc in jupyterlab 4.x for my personal use.

The way I could see it working would be to add a toolbar toggle button and add a config/setting so the user can include that feature or not. To achieve the display of the code cells, the toc extension would have to be modified. Here are the suggested modifications :

• tocitem.tsx : run an additional check (only in the case of a NotebookPanel) to see if the heading cellRef corresponds to a code cell. If it does, render a CodeMirror editor with the cell input content instead of :

<button>
    ...
</button>
<span className="jp-tocItem-content" title={heading.text} {...heading.dataset}>
    {heading.prefix}
    {heading.text}
</span>

• create a non editable CodeMirror editor component to display the code cell input and style it similarly to what's done in the old toc. Or, use the available jupyterlab API to build an editor.

[fcollonval]

For information, that specific feature was removed for two reasons:

• it brings trouble to make it accessible
• it have a huge performance cost for long notebook

The way forward is likely not to include that in the table of content but to bring either or both:

• a filtering method at the document level as the real value is to be able to quickly highly peculiar cells (based on tag for example)
• a kind of minimap

[krassowski]

Adding to the above, we could also consider:

• adding it back as opt-in (off by default) but instead of rendering the cells using a full CodeMirror editor, just show a lightweight placeholder (e.g. plain text, even without syntax highlighting)
• hosting an extension which implements the old feature in jupyterlab-contrib organization (and then we would accept PRs exposing API that is necessary for it - if any changes are needed).

[Rmarieta]

Thank you for the quick reply.

I will probably first go for publishing the extension in jupyterlab-contrib for simplicity and no changes are needed in the JupyterLab API as it is.

Regarding the huge performance cost, was it mostly due to having to build numerous code cell editors in the case of long notebooks or to the toc becoming populated with too much content ? Would opting for a more lightweight way to display the code solve that problem or not ?

[fcollonval]

Thanks a lot @Rmarieta for proposing your help

Regarding the huge performance cost, was it mostly due to having to build numerous code cell editors in the case of long notebooks or to the toc becoming populated with too much content ? Would opting for a more lightweight way to display the code solve that problem or not ?

Both: lots of DOM nodes and lots of editors (CM 6 used in Lab 4 help on that front).

---

I was thinking again about this. And a nice meaningful compact view of code cells that I would support to add to the ToC would be to display the cell code outline instead of the content; like you can do with LSP:

Example of type hierarchy in VS Code.

or

Same code but symbols tree on GitHub

[krassowski]

The LSP outline is high on priority list for LSP extension - but first need to finish porting it to 4.0. And could go straight to core if we want :)

Inspired by the above, one notebook-oriented approach would be to show one line for each cell and (optionally) each output (truncated to say 3 outputs):

- header 1
 - [ print(2 * 2) ]    # shows first line of code cell only, truncated to n characters
   - 4                 # shows first line of text output
 - [ pd.DataFrame(matrix) ]
   - DataFrame (10 x 10)
 - [ plot(x) ]
   - 🖼                # shows an icon representing image output but not the image itself

For me when analysing data this would be more useful than having code symbols - when navigating across large notebook I am usually interested in navigating to a specific visualisation/table I am working on rather than to a specific variable, because relevant variables are all proximal to the cell of interest (or imported) anyways.

[Rmarieta]

I agree that

Inspired by the above, one notebook-oriented approach would be to show one line for each cell and (optionally) each output (truncated to say 3 outputs):

 - [ print(2 * 2) ]    # shows first line of code cell only, truncated to n characters
   - 4                 # shows first line of text output
 - [ pd.DataFrame(matrix) ]
   - DataFrame (10 x 10)
 - [ plot(x) ]
   - 🖼                # shows an icon representing image output but not the image itself```

would be an interesting solution to integrate the code cell input/output content as part of the toc.

@krassowski I'm not so familiar with the LSP extension, are you talking about jupyterlab-lsp or the lsp-extension package ? And basically the idea would be to wait for that LSP outline implementation to be done to then rethink the toc around it and incorporate code cell outline ?

[krassowski]

are you talking about jupyterlab-lsp or the lsp-extension package

These two will hopefully converge in the future, I was in fact suggesting that it could go to the second one, skipping incubation in the former as outline is an additive low-risk feature.

the idea would be to wait for that LSP outline implementation to be done to then rethink the toc around it and incorporate code cell outline ?

The way I see it is that LSP outline would not give us the notebook structure needed for more notebook-oriented navigation. Maybe these should be two separate widgets or just separate subtrees in a single "navigation" panel. Maybe there are better ways to approach it - I am curious what @fcollonval envisioned.

[fcollonval]

@krassowski I like your proposal of having two ToCs as proposed in jupyter-lsp/jupyterlab-lsp#970