Any interest in a new message type for transient information?

[BoPeng]

This was originally posted to the jupyterlab issue tracker (jupyterlab/jupyterlab#4514) but I was told that the topic suits here better.

To my understanding right now a kernel can send messages such as display_data, stream, error, and execute_reply to the frontend, and all messages are displayed in the output area of cells. There are however scenarios where messages are meant to be transient and not part of the notebook. For example,

1. Status/Progress information. For calculations that take a long time to execute, the kernel might want to send some progress information to the frontend.
2. Debug information. A kernel might want to output more verbose information under certain debug mode. The information can be long and are not meant to be saved with the notebook.
3. Side effect of calculation. In addition to regular result, a kernel might want to output some information about the result. As an extreme example, after the execution of a=1 that does not produce any real output, the kernel might want to display the value of a as a side effect of the calculation.

All these can be achieved through Jupyter/JupyterLab extensions using customized comm channels but it is non-trivial to develop extensions and all extensions would use different approaches to display the information. I am therefore wondering if a new message type for transient information can be added. Basically,

1. A message would be similar to display_data, but with an additional title field to indicate what the message is about. The message was originally named kernel_info but it conflicts with the existing kernel_info message so an appropriate name is needed.
2. A kernel can send the message to ioPub without need for a customized comm.

Frontends can

1. Ignore this message if there is no UI element to display it (e.g. in case of classic jupyter)
2. Develop a new information-center/bulletin-board kind of UI to display the content of the message. The location of the UI is up to discussion but I have a more or less working branch of JupyterLab that adds an info panel as a tab to an inspector panel.

I do not know how difficult it is to add such a message type but I will be happy to contribute to the work if it is potentially acceptable.

[vidartf]

If I understand you correctly, this has no relation to the current kernel info messages (http://jupyter-client.readthedocs.io/en/stable/messaging.html#kernel-info), but would be a new message (maybe with another name to differentiate it)?

[BoPeng]

Yes, you are right. Let me think of a name. It is sort of multi-functional so perhaps transient-data or transient-info? The original post has been updated.

[jasongrout]

Note that there already is a transient part of a display_data message: http://jupyter-client.readthedocs.io/en/latest/messaging.html#display-data

[BoPeng]

New in 5.1, no wonder I do not know it before.

The only transient key currently defined in Jupyter is display_id

Are we free to send whatever information in this transient dict? I mean, if a kernel sends

transient: {
    'title': 'a',
    'data': '1'
}

to the frontend, JLab would simply ignore it, right? Is there a need to formally change the protocol if I submit a PR to handle other fields of transient for the purposes described in this ticket?

[jasongrout]

I think any other field is undefined at this point. CC also @minrk and @rgbkrk who also helped develop the current spec.

[BoPeng]

Undefined but can be passed without causing any error, right? That means if we handle other fields here, JLab would be able to send transient information to the information/inspection panel that I envisioned, without the need to change protocol and without breaking classic Jupyter.

[jasongrout]

Undefined means that the behavior is undefined, but it should be passed along too. In the future keys might be defined that conflict, though.

[takluyver]

My understanding is that the transient namespace is similar to metadata: you're free to use it, but it's recommended that you use the name of your extension (or whatever project is defining the meaning) as a namespace, to minimise the risk of clashes.

[rgbkrk]

@ivanov, @yuvipanda, and I started work on a new message type that is effectively transient, called the Resource Info Request in jupyter/jupyter#264. It's finely scoped to live metrics collection. Pointing this out mostly to get an idea of how wide protocol changes end up getting worked on. I definitely hear you about comms -- they get hairy because of their looseness. It's super great that you want to make a formal message to use!

As for the message type you're proposing, I think this makes sense as transient_display_data since it's effectively a display data payload yet is not intended to be stored in a notebook or otherwise. I guess the open question is how a frontend interprets where to "place" it. Do we have a key for "target" or something, where effectively the intent can be stated as "output", "panel", or something else?

My typical bar for creating a new message type is that it's scoped to a particular area. In the cases, like this one, where it's generic the big asks are going to be that the spec is formalized as much as possible and that metadata doesn't become the escape hatch for fields that should be specified a bit more rigorously within the main content payload.

Any frontend will drop messages they don't understand (at least, they should). In practice anything not understood is effectively transient, so you should definitely feel free in experimenting. 😄

---

As an aside, since it appears there was some confusion about the transient field -- the intent is to not persist what is in transient within the notebook document from an output. Frontend gets access, it just doesn't store it.

[takluyver]

If it's going to work similarly to a display_data message, would it make sense to make it a display_data message with some metadata indicating that it's transient, rather than a new message type?

That would mean that frontends unaware of the new functionality would treat it as a normal display output, rather than ignoring it. That may or may not make sense, depending on your use case.

[BoPeng]

I agree with @takluyver , I was thinking of placing data inside transient

display-data: {
    'data': null,
    'transient': {
        'title': 'what data is',
        'data': 'transcient_data',
    }
}

but then I think it would be better to have something like

display-data: {
    'data':  'transcient data',
    'transient': {
        'title': 'what data is',
        'target': 'panel',
    }
}

In this way, whereas JLab can send the transient data to a dedicated panel, Jupyter and other frontends can handle transient data as regular display-data.

[BoPeng]

@rgbkrk , I though of using this type of transient message to display metrics and I actually plan to use it to display such information (e.g. the CPU/MEM usage of %taskinfo magic). My concern was that there are so many different types of "metrics" so unless they are defined really rigorously (like what you did in jupyter/jupyter#264), it is easier just to let the kernel decide how to display such information (e.g. a figure generated by the %taskinfo magic).

Where to display the transient information certainly depends on what types of information you would like to display. Whereas the metrics could be displayed, for example, as barcharts in a status bar somewhere, I plan to allow arbitrary display-data to be displayed in a dedicated side panel. The following is an illustration of the idea from my JLab info-panel fork where an info panel is placed as a tab to an inspector side panel, and is used to, among many possibilities, preview variables in tabular format.

[rgbkrk]

If you do this on the current display data, even with some metadata saying the overall message / content is intended to be transient, any consumer of your library that uses the classic notebook (or really any other frontend that isn't up to date with this new thing) will end up writing a lot to disk that you didn't intend to. You should definitely make a new message type even if it's the same type of payload as display_data.

[jasongrout]

This would be similar to how we made an update_display_data message, which is not interpreted as a display_data message: http://jupyter-client.readthedocs.io/en/latest/messaging.html#update-display-data

[BoPeng]

I agree that a new message type makes more sense than hacking display_data. I mean, although display_data has a transient field, it is meant to assist the display of data, not to have its own data as in transient: {'data': 'transient_data' }. Using transient: {'target': 'panel'} to redirect data also has its own problems as @rgbkrk pointed out.

Adding a new message type sounds to me like a fundamental change that needs long and careful contemplation but I created a PR with a draft definition anyway. 😄

[takluyver]

Maybe using update_display_data would work for this? It looks like if you send a display ID that the frontend doesn't know about, it will discard the output:

https://github.com/jupyter/notebook/blob/5cfba9cb24a6d06b2794dbbcda6b211477a270f1/notebook/static/notebook/js/outputarea.js#L612-L616

You could even make it possible to target an actual display ID and have an updating area just as the output from one cell.

[BoPeng]

Yes conceptually speaking a frontend can create a global output area somewhere with a special display_id (e.g. transient) and allow all kernels to update it with update_display_data messages. This sounds reasonable although

1. Currently _display_id_targets are restricted to the current cell. It could be initialized with an global target area though.
2. The update_display_data message does not allow the setting of a title, which is important for this message type.
3. With a single area/display_id it can be tricky to append messages to displayed ones (e.g. append step 2 completed after step 1 completed).

[jasongrout]

The update_display_data message does not allow the setting of a title, which is important for this message type.

In general, display_data messages don't support titles. To me, a title sounds like a metadata argument, which doesn't need to be specified in the message content.

With a single area/display_id it can be tricky to append messages to displayed ones (e.g. append step 2 completed after step 1 completed).

This is sounding more like an ipywidget Output widget, in the vein of https://github.com/jupyter-widgets/jupyterlab-sidecar.

[jasongrout]

(CC @SylvainCorlay)

[rgbkrk]

It looks like if you send a display ID that the frontend doesn't know about, it will discard the output

While frontends (can) ignore a display ID it doesn't know about, encouraging this alters the meaning of the spec for update display messages. I'm not opposed, my sense is that we'd want to be explicit about it in our documentation if it would become something to expect.

Currently _display_id_targets are restricted to the current cell. It could be initialized with an global target area though.

Do you mean restricted to the current notebook? You're able to use them across cells:

The update_display_data message does not allow the setting of a title, which is important for this message type.

In general, display_data messages don't support titles. To me, a title sounds like a metadata argument, which doesn't need to be specified in the message content.

I definitely agree with @jasongrout on that one -- seems like it could be in the metadata

With a single area/display_id it can be tricky to append messages to displayed ones (e.g. append step 2 completed after step 1 completed).

Yeah, it definitely requires sending an overall payload to update rather than individual changes. @minrk and I went with this to make it simple and consistent (and at the time there weren't any output types that were JSON backed where update diffs made sense).

[BoPeng]

Yeah, it definitely requires sending an overall payload to update rather than individual changes. @minrk and I went with this to make it simple and consistent (and at the time there weren't any output types that were JSON backed where update diffs made sense).

This is unrealistic because you are asking kernels to keep a copy of all previous status/progress/debug messages and send them altogether each time. The kernel will either have to maintain a global variable or send a variable to each function that might send an update, and passing and rendering of long content can also cause problems.

[SylvainCorlay]

Thanks for the ping @jasongrout. I think we can somewhat relate this to #373 which is about other streaming message types than stdout / stderr.

[SylvainCorlay]

On the subject of displaying things in other parts of the UI (outside of the notebook), you may be interested in the sidecar widget for jupyterlab.

This allows to have transient displayed data which is not part of the notebook state.

To display something in a sidecar with a prescribed title, you can do.

from sidecar import Sidecar

sc = Sidecar(title='Some nice title')
with sc:
    print('hello, world')
    display(rich_output)

@BoPeng does this correspond to your use case?

[SylvainCorlay]

IMO, this kind of approach has the advantage of not requiring a new message type.

[BoPeng]

@SylvainCorlay This message type is designed for kernels to send any transient information to the frontend. It is meant to be

1. transient, so its content can be silently overwritten (unless the frontend keeps a history) by other transient messages. This is different from the behavior of the Sidecar panel as far as I can tell.
2. very general. I summarized some use cases here but nothing prevents a kernel from sending other transient information (e.g. CPU/MEM metrics) through this message type.
3. available to all kernels, not limited to Python kernels for which Sidecar is designed. (Correct me if I am wrong).
4. transparent to users, meaning that in most cases (e.g. in cases of spontaneous status/progress update from kernels) there is no need for user intervention , and in other cases (e.g. user requested transient information) user only need to specify "show this in information panel, not in my notebook". Sidecar is nice but is too complex to use IMHO.

I have seen other replies that suggest some overlapping use cases and alternative implementations. I think one of the goals of the transient_display_data message type is to unify some common use cases that have been implemented in complex ways (e.g. customized comm) with different UIs (e.g. various side panels, widgets) to make it easier for developers (of kernels) to show, and for users to see such information.

[SylvainCorlay]

Quick precision: comms and widgeta are not specific to Python. The example with the sidecar shows that using the current protocol, you can get capture outputs and display messages and display them in a separate part of the user interface, and not keep it as part of the state of the document.

I don't thonk that comms are especially complex. They are well suited for this kind of usecase. The widgets layer makes it even simpler to use.

I fear that we inflate the kernel protocol to reinvent what the comms and widgets protocol solve well.

[BoPeng]

How do you envision displaying this type of info?

It is up to the frontend to decide how to display them but here is what I have in mind for JLab:

1. A dedicated side panel (potentially as a tab of an inspector panel) with a title to indicate what the message is about and an output area similar to cell outputarea. These could be implemented as a readonly "cell".

2. Although the messages could be concatenated (as if in a terminal or bulletin board), or saved in a history (e.g. with navigation buttons), I think in generally there is no strong need to save history of transient messages.

3. However, there are cases (e.g. multiple debug messages sent during the execution of a cell) where messages that belong to the same group should be kept together. I therefore proposed a "title" and "append" mechanism where 1) by default new messages replaces existing one. 2) If append=true and the current display has the same title, the new message is appended to the old one.

This message shows a screenshot of proposed interface where an info panel is added as a tab to an inspector and displays the result of a %preview magic that shows the content of a dataframe.

[minrk]

It is up to the frontend to decide how to display them

It can't be quite that loose. the spec has to make some assertions about how they should be displayed (i.e. among outputs vs somewhere else). It sounds like you mean that these should never appear in the cells that produce them, so while the UI may decide how "somewhere else" works, it is at least nowhere in the notebook itself. That does point to it being a new message, because if it were treated as a display_data message it would not only be saved when it shouldn't be (generally not a big deal), it would appear in the wrong place (a bigger deal).

It also sounds like the two cases of a generic "state" message that gets updated and replaced versus a debug/transient log of regular display-data messages. I don't think the append=True/False method on a single message is a good fit for differentiating these two, since these are two very different use cases and behaviors, with different message patterns, requirements, etc. Maybe this should be reserved for the dedicated 'transient log', and the resource_info_request/reply proposal can serve the sticky state message area?

The log approach fits the sidecar model very well, with one missing piece: there isn't currently a way without widgets to create a new output area not part. sending outputs with a specified parent will route them to the appropriate area using all of the existing mechanisms without any need for a new message. What's missing, and what widgets are used for, is creating that output area in the first place. If there were an easier way to create a new output area, then I think this case would be totally covered.

[BoPeng]

@SylvainCorlay I don't think that comms are especially complex. They are well suited for this kind of usecase. The widgets layer makes it even simpler to use

comm + widgets can be used to address all the usecases, but not necessarily "easily". To the very least, an extension would have to be developed and installed (at least for JLab) which is an extra burden to both developers and users, and there can be many different implementations (sidepanel, widgets etc) for similar purposes.

[BoPeng]

these should never appear in the cells that produce them, so while the UI may decide how "somewhere else" works, it is at least nowhere in the notebook itself.

Exactly. These messages should not be saved and displayed with the notebook.

two cases of a generic "state" message that gets updated and replaced versus a debug/transient log of regular display-data messages

I do not think a "log of regular display-data messages" model fits the use cases of transient messages. To my understanding, transient messages are used to assist interactive data analysis where the content of messages quickly become irrelevant or outdated. For example, the status/progress information of a cell would be useless as soon as the execution is completed; status enquires are valid for the time of enquiry only; the debug information can be tossed away as soon as users examine them, and preview of variables would be outdated as soon as users modify the code and re-execute the cell.

So in my opinion transient messages are largely outdated when the user starts to execute the next cell. That is to say, whereas multiple transient messages produced from the execution of a cell (e.g. multiple debug messages) should be kept together, there is no need to keep previous messages in a log.

[rgbkrk]

I fear that we inflate the kernel protocol to reinvent what the comms and widgets protocol solve well.

We should use comms as a last resort. Comms were intended as an escape hatch to make progress on the widgets years ago without needing to establiash changes to the protocol. This is why they have so few constraints on what they can do.

I don't thonk that comms are especially complex. They are well suited for this kind of usecase. The widgets layer makes it even simpler to use.

Comms are very understandable for each of us. Their layer is pretty well established. It certainly can be done. That's not what's being questioned.

As @BoPeng points out, they require writing an extension. Which means users have to install it to use it. When a protocol gets built in, frontends can implement functionality in the way that makes sense for them (reminder -- there's not just jupyterlab and nteract -- there's console, nbconvert, papermill, cocalc, kaggle, and many others).

On top of that, there's always the sticking point of non-backwards compatibility with the widgets. The kernel and the frontend have to stay in sync. You can't move the frontend forward without moving the kernel forward. As an operator, this is problematic because sometimes you need a kernel that matches old versions of libraries (even if trailing by only a week). This request for a protocol addition is a way to make something formal that we can adapt and version along with the rest of the message spec.

[rgbkrk]

So in my opinion transient messages are largely outdated when the user starts to execute the next cell. That is to say, whereas multiple transient messages produced from the execution of a cell (e.g. multiple debug messages) should be kept together, there is no need to keep previous messages in a log.

Just a note on the way we've been handling async jobs within the notebook. For a %%presto (and %%hive) magic is to allow the user to run the job in the background (on a thread) while showing a progress bar. Once it's finished it puts the results in a DataFrame under a variable the user specified.

We do clear the progress bar out once it's done. One important part about how this works is that if this notebook was running with a non-interactive frontend (papermill), we are still writing these updates to disk which means that when viewed in a notebook viewer they can see the progress. This does mean we're storing it in the notebook (with update display data). I could imagine us having other transient information that shouldn't persist and would only be useful in a live session. 🤔

[rgbkrk]

Maybe this should be reserved for the dedicated 'transient log', and the resource_info_request/reply proposal can serve the sticky state message area?

One nice thing about the resource info proposal is that it is organized into namespaces. Does this approach what you're hoping for in a title? Am I missing something else, like that you're wanting a dedicated pane to be shown with the title?

[BoPeng]

async jobs within the notebook. ... run the job in the background (on a thread) while showing a progress bar.

This is clearly out of the scope of transient messages. The SoS kernel also has something similar (remote tasks) but instead of a dedicated "live" progress bar, users are asked to use magics such as %taskinfo to explicitly query the status of tasks. The output of that magic clearly should be transient.

[BoPeng]

One nice thing about the resource info proposal is that it is organized into namespaces. Does this approach what you're hoping for in a title?

I will let @minrk answer your question. I do think resource info is a nice thing to have and it is great that it has formal definitions and namespaces, and I would suggest a third tab in the inspector panel that I proposed to display such information. That is to say,

1. the "hint" tab is the original "inspector". It can be updated with 'Shift-Tab' or automatically while typing.
2. the "info" tab is the information center for transient messages.
3. the "resource" tab can have live monitors updated by resource info messages.

All three tabs display "transient" information during interactive data analysis and can be grouped together in an "inspector" panel.

[rgbkrk]

The SoS kernel also has something similar (remote tasks) but instead of a dedicated "live" progress bar, users are asked to use magics such as %taskinfo to explicitly query the status of tasks. The output of that magic clearly should be transient.

Oh cool, this is good to know about.

[SylvainCorlay]

I fear that we inflate the kernel protocol to reinvent what the comms and widgets protocol solve well.

We should use comms as a last resort. Comms were intended as an escape hatch to make progress on the widgets years ago without needing to establiash changes to the protocol. This is why they have so few constraints on what they can do.

I disagree with that statement.

comm + widgets can be used to address all the usecases, but not necessarily "easily". To the very least, an extension would have to be developed and installed (at least for JLab) which is an extra burden to both developers and users, and there can be many different implementations (sidepanel, widgets etc) for similar purposes.

Adding a message type also requires extra development, at a deeper level. Also, JupyterLab was built from the start as a collection of plugins. Installing more should not be considered an issue. Also, which plugins are elected to be part of the base installation is not set in stone.

[BoPeng]

Adding a message type also requires extra development, at a deeper level. Also, JupyterLab was built from the start as a collection of plugins. Installing more should not be considered an issue.

I believe transient display data is useful to many kernels. We do not see many plugins of this type because it does not worth the trouble to develop plugins for such "simple" purposes, and these plugins are not welcome anyway, just imagine 10 widgets each displaying transient information from their respective kernels.

So whereas "installing more plugins" might not be a problem, limiting kernels from displaying useful transient information is a problem and having multiple plugins with similar purposes would be a bigger problem. The benefits of solving the problem "at a deeper level" are therefore providing a unified solution for the display of transient data and saving kernel developers from the need to create specialized plugins.

[SylvainCorlay]

I don't think that there are any intrinsic limitation from using the widgets API for communicating between kernels and front ends. You can have a single UI component driven by jupyter widgets for multiple kernel instances and languages. Starting with widgets can also help narrow what new messages may be needed in the future should they be added to the protocol. In the mean time you can already do a lot with comms and the widgets protocol. Just for the record, it took about a year for the `comm_info` request / reply messages to be included in the spec.

…

On Wed, May 9, 2018, 23:04 Bo ***@***.***> wrote: Adding a message type also requires extra development, at a deeper level. Also, JupyterLab was built from the start as a collection of plugins. Installing more should not be considered an issue. I believe transient display data is useful to many kernels. We do not see many plugins of this type because it does not worth the trouble to develop plugins for such "simple" purposes, and these plugins are not welcome anyway, just imagine 10 widgets each displaying transient information from their respective kernels. So whereas "installing more plugins" might not be a problem, limiting kernels from displaying useful transient information is a problem and having multiple plugins with similar purposes would be a bigger problem. The benefit of solving the problem "at a deeper level" is therefore providing a unified solution for the display of transient data and saving kernel developers from the need to create specialized plugins. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#376 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACSXFmuouY5NrfPobXvFs6HMIYI6WOzQks5tw0vagaJpZM4TytF6> .

[BoPeng]

You can have a single UI component driven by jupyter widgets for multiple kernel instances and languages. ... In the mean time you can already do a lot with comms and the widgets protocol.

Kernels are maintained by different developers. Developing a third-party plugin with a widget and customized comm for multiple kernels is unrealistic. Adding a standard UI with a customized comm that could be replaced with a standard message type (in a year?) does not sound like a good idea either. Given the two-body (message/UI) nature of this message type, I believe some consensus on at least the need for a general solution is needed to push a UI forward. I do not know how such a consensus can be reached (a developer meeting? A poll somewhere?) but I guess I have expressed my ideas clear enough here and will just wait.

[rgbkrk]

In practice, for jupyter, consensus only gets reached through merging pull requests. Developer meetings are good at establishing what's likely to be merged in some future date. Without a prototype, people are commenting on the idea and it's hard to gain a sense of who is in approval as well as what changes are really needed to be made.

@BoPeng -- here's what you need to do to get your idea off the ground and seriously evaluated:

• Write up the new spec for this transient_display_data message as a PR to jupyter_client
• Open up a PR to ipykernel and jupyterlab that demonstrates how this is used
  • Note: You'll get more buy-in if you implement it in other frontends, as you'll end up with more backers to your idea

[BoPeng]

here's what you need to do to get your idea off the ground and seriously evaluated:
Write up the new spec for this transient_display_data message as a PR to jupyter_client

Already done (#378 (comment)) although this is just an initial proposal and will likely change.

Open up a PR to ipykernel and jupyterlab that demonstrates how this is used

Just did it for JLab: jupyterlab/jupyterlab#4550

Note: You'll get more buy-in if you implement it in other frontends, as you'll end up with more backers to your idea

Let me start from JLab as I am actively working on it to port my SoS kernel over, but I certainly need/welcome more advocates.

[BoPeng]

Just to add that another nice application of this message type would be source code analysis (as discussed here). Basically, many languages have their own linter tools and it would be nice for the kernel to send linter messages to the frontend during interactive data analysis.

[BoPeng]

Whereas a display_data-like message can pass all types of information, it is a bit troublesome to pass stream-like message for simple text-based information such as progress, log, and error messages. My understanding is that a stream message with content

{
    'name' : stdout,
    'text' : 'step 1 is done',
}

can be translated to a transient_display_data message with content

 {
    'data' :  {
        'application/vnd.jupyter.stdout': 'step 1 is done'
    }
}

but the use of application/vnd.jupyter.stdout is not obvious and should at least be documented for this application.

Edit: application/vnd.jupyter.stdout works only for JupyterLab. Jupyter handles streams directly and does not understand application/vnd.jupyter.stdout. So the only reliable way to handle stream would be using text/plain for stdout, and text/html and some html conversion for stderr.

[BoPeng]

Just a quick update: an UI to display this message type has been proposed for JupyterLab (jupyterlab/jupyterlab#4879).

[BoPeng]

@BoPeng -- here's what you need to do to get your idea off the ground and seriously evaluated:
Write up the new spec for this transient_display_data message as a PR to jupyter_client

Done.

Open up a PR to ipykernel and jupyterlab that demonstrates how this is used

A UI is proposed for JupyterLab (jupyterlab/jupyterlab#4879) with a test extension to create and send transient_display_data message to the UI. There is no side panel for classic Jupyter so I have implemented this message to display the data in the side panel of SoS Notebook. What needs to be done for ipykernel?