Xeus incorporation

[SylvainCorlay]

We propose to make Xeus an official Jupyter subproject.

Votes for approval:

• @afshin
• @blink1073
• @Carreau
• @damianavila
• @ellisonbg
• @fperez
• @ivanov
• @jasongrout
• @jhamrick
• @minrk
• @mpacer
• @parente
• @rgbkrk
• @Ruv7
• @SylvainCorlay
• @takluyver
• @willingc

[minrk]

I think this is great, and happy to include Xeus. The only downside I see to including xeus is that incorporating it into Jupyter makes the vibrant third-party kernel community look a little less great, as we bring one of the best bits in as first-party :).

I know naming things is hard, but jupyter-native seems like a confusing name. What's it native to? It's not native to Jupyter, etc. I know it's native to the processor because it's compiled, but the phrase "jupyter-native" doesn't communicate that as well, and the term "native kernels" also doesn't seem a good fit for describing kernels that are compiled. Can we not just keep the name xeus? We could use jupyter-xeus for the GitHub org if folks want to make the Jupyter subproject relationship clear, but still call everything xeus.

[SylvainCorlay]

I know naming things is hard, but jupyter-native seems like a confusing name. What's it native to? It's not native to Jupyter, etc. I know it's native to the processor because it's compiled, but the phrase "jupyter-native" doesn't communicate that as well, and the term "native kernels" also doesn't seem a good fit for describing kernels that are compiled. Can we not just keep the name xeus? We could use jupyter-xeus for the GitHub org if folks want to make the Jupyter subproject relationship clear, but still call everything xeus.

I am not opinionated at all about that name. We thought of "native" because it is mostly compiled code. I am also OK with jupyter-xeus...

[jasongrout]

Point of order: I just added a voting checklist to the OP. Now we have three ways to vote, the emojis, the github approval system, and the checklist. I've updated the checklist to merge each of these results.

• emoji: Easiest to vote, but provides no audit trail of the vote and votes can be silently removed
• checklist: Easy to vote, pings all steering committee members. Can record votes for others if they approved elsewhere or in other ways, and there is an audit trail of who recorded the vote, though the audit trail (the comment history) is not very easy to see (so perhaps is not so useful in practicality). Can be used on issues and PRs.
• github approvals: Nicely summarized in issue and in sidebar, has very understandable audit trail. Hardest to vote and OP cannot vote. Reviews can be requested, but seems like we can't request the entire steering council easily. Can only be used on PRs.

[jasongrout]

I know naming things is hard, but jupyter-native seems like a confusing name. What's it native to? It's not native to Jupyter, etc. I know it's native to the processor because it's compiled, but the phrase "jupyter-native" doesn't communicate that as well, and the term "native kernels" also doesn't seem a good fit for describing kernels that are compiled. Can we not just keep the name xeus? We could use jupyter-xeus for the GitHub org if folks want to make the Jupyter subproject relationship clear, but still call everything xeus.

I agree that jupyter-xeus would be a better name for the Github org, or even just xeus.

[saulshanabrook]

I opened a discussion earlier today on the Jupyter Debugger repo about using xeus Python kernel vs ipykernel (jupyterlab/debugger#274). At the JupyterLab dev meeting, some folks suggested I move the discussion here to encourage more participation and visibility.

The impetus is that we (@Quansight) have a client who wants us to help move forward the debugging support in JupyterLab. Currently, if they were to begin adopting it with a Python kernel, they would have to switch to xeus-python instead of ipykernel, because xeus-python has support for the debugging protocol and ipykernel does not. So the question is, should we put our resources towards making sure xeus-python works for them or on adding debugger support for ipykernel?

More broadly, in the JupyterLab context, when Jupyter debugger is stable and we start encouraging use of it, we will have the same question. Of course technically we can tell users that if they want debugging support they have to switch kernels and if they want any features that ipykernel has that are not supported in xeus-python, then they should stick with that, but that isn't a satisfying answer from a UX perspective. When a new user comes to JupyterLab, eventually we would like them to be able to start debugging in Python with minimal friction.

My understanding is that adding debugging support to ipykernel would require a significant refactor. So my hope is to get a sense from folks who are currently, or have been, active in ipykernel and those in xeus-python (cc @minrk @ivanov @Carreau). whether we can put together a plan to either:

1. End up with xeus-python as the default python kernel
2. Refactor ipykernel so it can support the debugging protocol

I don't have a horse in the race, but just want to understand which would be useful for us to allocate resources towards, as client needs come up that affect the Python kernel.

On a meta level, I am open to any feedback or suggestions on where this conversation should be held and how to move it forward in a way that is inclusive and transparent.

EDIT: Here are some links to

• Jupyter Debugging protocol: Debug Protocol Overview jupyterlab/debugger#64 (comment)
• xeus-python implementation of the debugging protocol: https://github.com/QuantStack/xeus-python/blob/master/src/xdebugger.cpp

I am not sure what it would take to make xeus-python feature-complete with ipykernel. Someone mentioned something about magics offhand, but I personally have never used xeus-python and am not sure about its status.

[meeseeksmachine]

This pull request has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/ipykernel-vs-xeus-kernel-discussion/2877/1

[SylvainCorlay]

Thanks for opening this @saulshanabrook although I don't think that the JEP for incorporating xeus is the right place because for this. The JEP is not making a statement about an official recommended Python kernel, and it covers a broader set of projects, including xeus itself (the C++ library), xeus-cling and companion projects.

[saulshanabrook]

@SylvainCorlay Could you make a recommendation for where we should discuss this then?

[SylvainCorlay]

@SylvainCorlay Could you make a recommendation for where we should discuss this then?

Sure, I think that both ipykernel or xeus-python are fine. Although it seems you just opened issues and threads on all channels already

• discourse
• the mailing list
• twitter
• the xeus-python repository
• the debugger repository
• the ipykernel repository
• and here.

so I would just pick one of them and close the others!

[saulshanabrook]

Although it seems you just opened issues and threads on all channels already (discourse, xeus-python, debugger, ipykernel and here) so I would just pick one of them and close the others!

OK since I opened an issue on the debugger already, we can have the conversation over there: jupyterlab/debugger#274

[SylvainCorlay]

OK since I opened an issue on the debugger already, we can have the conversation over there: jupyterlab/debugger#274

OK thanks @saulshanabrook

[willingc]

It may make sense to remove reference to the debugger in the JEP. I was a bit surprised by the wording since it seems to imply movement from ipykernel to the xeus-python kernel. If this is not the case, I think this would read more clearly by removing the following part, which does not seem critical to the JEP itself. Alternatively, I would be more explicit about what is meant by the core library and where ipykernel fits in relationship to xeus-python.

Beyond the xeus-based kernels, there is a new push in the project motivated by the development of the JupyterLab visual debugger. We think that the core library is fairly complete already and that most of the work will be done in xeus-based language kernels such as xeus-cling and xeus-python.

[willingc]

I approve conditionally on clarification re: ipykernel and xeus-python in core library of debugger or removal of that wording.

[SylvainCorlay]

@willingc I have modified the wording into

Beyond the xeus-based kernels, there is a new push in the project motivated by the development of the JupyterLab visual debugger. We think that the core library is fairly complete already and that most of the work already done will apply to other xeus-based language kernels such as xeus-cling.

Are you fine with this?

[willingc]

Pretty much @SylvainCorlay. Perhaps clarify which core library. The visual debugger core library or xeus core library. Thanks!

[SylvainCorlay]

Pretty much @SylvainCorlay. Perhaps clarify which core library. The visual debugger core library or xeus core library. Thanks!

Done. I meant the core xeus library (not xeus-python). The front-end debugger UI will work with any kernel supporting the protocol.

[Carreau]

Can (Could) xeus-python use IPython as an execution engine ? What needs to change for this to happen ? My guess is that users do not really care about the ipykernel features which are super-minimal but the IPython features (magics, tracebacks, etc...)

Voted +1 in principle about adopting xeus.

[SylvainCorlay]

Can (Could) xeus-python use IPython as an execution engine ?

Probably, although it would probably make the codebase more complicated, and may not be the best approach to reach full feature parity (cf second part of the answer).

(magics, tracebacks, etc...)

We already support rich tracebacks, autocompletion, rich mimetype rendering, widgets, but not magics yet.

• although we have looked into enabling all IPython magics in xeus-python when IPython is present in the environment.
• I would like to have two levels of magics in xeus-based kernels: language-agnostic magics that we could implement once for any kernel, and kernel-specific magics such as the IPython ones.

[blink1073]

Given that we have 11 approvals from SC members and no disagreements, I propose we merge this tomorrow. We can continue discussions about ipykernel vs. xeus-kernel and debugging on other channels.

[Carreau]

We already support rich tracebacks, autocompletion, rich mimetype rendering, widgets, but not magics yet.

Let me rephrase then; I'm concern about the difficulty to maintain similar behavior in two different project. For example current work in IPython to get better stacktraces: ipython/ipython#11827

I think there is a good case to pull some of the functionalities into a ipython_core that could be reuse and guaranty identical behavior between IPython/Xeus-Python(/maybe xonsh)

But all of that is an implementation detail.

[SylvainCorlay]

I think there is a good case to pull some of the functionalities into a ipython_core that could be reuse and guaranty identical behavior between IPython/Xeus-Python(/maybe xonsh)

Yes - similarly, many libraries meant to be use in Jupyter make use of IPython APIs (for example display, get_ipython) I was looking at a common denominator that could be defined as an abstract API... This is not quite about converging on behavior though.

[rgbkrk]

Belated 👍