Notebook 7, ipykernel 6.27, ipython 8.17

[mkoeppe]

This is now based on JupyterLab. There are many changes in dependencies.

We are trying to use platform-independent wheels when available, to avoid any Node.JS activity during our build.
Hence we remove our nodeenv, nodejs, jupyter_packaging, hatch_nodejs_version packages.

There is trouble on the horizon regarding our model of building everything from source: The latest versions of jsonschema have switched from pyrsistent to a Rust-based package.
We use the newest versions that don't pull in the build dependency on the Rust compiler.

• Resolves Make jupyterlab a standard package, update notebook to 7.x #24904
• Resolves Meta-ticket: Transition to JupyterLab #30399
• Resolves Update ipykernel to 6.13.0 - reintroduces psutil #33772

📝 Checklist

• The title is concise, informative, and self-explanatory.
• The description explains in detail what this PR is about.
• I have linked a relevant issue or discussion.
• I have created tests covering the changes.
• I have updated the documentation accordingly.

⌛ Dependencies

• Depends on Upgrade ipython to 8.17.x #35251 (merged here)

[mkoeppe]

sage -t --long --random-seed=286735480429121101562228604801325644303 sage/interacts/test_jupyter.rst
**********************************************************************
File "sage/interacts/test_jupyter.rst", line 9, in sage.interacts.test_jupyter
Failed example:
    from ipywidgets.widgets.tests.utils import setup_test_comm
Exception raised:
    Traceback (most recent call last):
      File "/sage/src/sage/doctest/forker.py", line 709, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/sage/src/sage/doctest/forker.py", line 1144, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.interacts.test_jupyter[0]>", line 1, in <module>
        from ipywidgets.widgets.tests.utils import setup_test_comm
      File "/sage/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/ipywidgets/widgets/tests/utils.py", line 50, in <module>
        orig_comm = ipykernel.comm.comm.BaseComm
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    AttributeError: module 'ipykernel.comm.comm' has no attribute 'BaseComm'
**********************************************************************
File "sage/interacts/test_jupyter.rst", line 10, in sage.interacts.test_jupyter
Failed example:
    setup_test_comm()
Exception raised:
    Traceback (most recent call last):
      File "/sage/src/sage/doctest/forker.py", line 709, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/sage/src/sage/doctest/forker.py", line 1144, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.interacts.test_jupyter[1]>", line 1, in <module>
        setup_test_comm()
        ^^^^^^^^^^^^^^^
    NameError: name 'setup_test_comm' is not defined
**********************************************************************
sage -t --long --random-seed=286735480429121101562228604801325644303 sage/repl/ipython_kernel/interact.py
**********************************************************************
File "sage/repl/ipython_kernel/interact.py", line 12, in sage.repl.ipython_kernel.interact
Failed example:
    from ipywidgets.widgets.tests.utils import setup_test_comm
Exception raised:
    Traceback (most recent call last):
      File "/sage/src/sage/doctest/forker.py", line 709, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/sage/src/sage/doctest/forker.py", line 1144, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.ipython_kernel.interact[0]>", line 1, in <module>
        from ipywidgets.widgets.tests.utils import setup_test_comm
      File "/sage/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/ipywidgets/widgets/tests/utils.py", line 50, in <module>
        orig_comm = ipykernel.comm.comm.BaseComm
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    AttributeError: module 'ipykernel.comm.comm' has no attribute 'BaseComm'
**********************************************************************
File "sage/repl/ipython_kernel/interact.py", line 13, in sage.repl.ipython_kernel.interact
Failed example:
    setup_test_comm()
Exception raised:
    Traceback (most recent call last):
      File "/sage/src/sage/doctest/forker.py", line 709, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/sage/src/sage/doctest/forker.py", line 1144, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.ipython_kernel.interact[1]>", line 1, in <module>
        setup_test_comm()
        ^^^^^^^^^^^^^^^
    NameError: name 'setup_test_comm' is not defined
**********************************************************************
sage -t --long --random-seed=286735480429121101562228604801325644303 sage/repl/ipython_kernel/widgets.py
**********************************************************************
File "sage/repl/ipython_kernel/widgets.py", line 11, in sage.repl.ipython_kernel.widgets
Failed example:
    from ipywidgets.widgets.tests.utils import setup_test_comm
Exception raised:
    Traceback (most recent call last):
      File "/sage/src/sage/doctest/forker.py", line 709, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/sage/src/sage/doctest/forker.py", line 1144, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.ipython_kernel.widgets[0]>", line 1, in <module>
        from ipywidgets.widgets.tests.utils import setup_test_comm
      File "/sage/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/ipywidgets/widgets/tests/utils.py", line 50, in <module>
        orig_comm = ipykernel.comm.comm.BaseComm
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    AttributeError: module 'ipykernel.comm.comm' has no attribute 'BaseComm'
**********************************************************************
File "sage/repl/ipython_kernel/widgets.py", line 12, in sage.repl.ipython_kernel.widgets
Failed example:
    setup_test_comm()
Exception raised:
    Traceback (most recent call last):
      File "/sage/src/sage/doctest/forker.py", line 709, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/sage/src/sage/doctest/forker.py", line 1144, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.ipython_kernel.widgets[1]>", line 1, in <module>
        setup_test_comm()
        ^^^^^^^^^^^^^^^
    NameError: name 'setup_test_comm' is not defined
**********************************************************************
1 item had failures:
   2 of   3 in sage.repl.ipython_kernel.widgets
    [99 tests, 2 failures, 0.48 s]
sage -t --long --random-seed=286735480429121101562228604801325644303 sage/repl/ipython_kernel/kernel.py
**********************************************************************
File "sage/repl/ipython_kernel/kernel.py", line 48, in sage.repl.ipython_kernel.kernel.SageKernel.__init__
Failed example:
    from sage.repl.ipython_kernel.kernel import SageKernel
Expected nothing
Got:
    0.00s - Debugger warning: It seems that frozen modules are being used, which may
    0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
    0.00s - to python to disable frozen modules.
    0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
**********************************************************************
1 item had failures:
   1 of   3 in sage.repl.ipython_kernel.kernel.SageKernel.__init__
    [12 tests, 1 failure, 0.93 s]
sage -t --long --random-seed=286735480429121101562228604801325644303 sage/repl/ipython_kernel/widgets_sagenb.py
**********************************************************************
File "sage/repl/ipython_kernel/widgets_sagenb.py", line 11, in sage.repl.ipython_kernel.widgets_sagenb
Failed example:
    from ipywidgets.widgets.tests.utils import setup_test_comm
Exception raised:
    Traceback (most recent call last):
      File "/sage/src/sage/doctest/forker.py", line 709, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/sage/src/sage/doctest/forker.py", line 1144, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.ipython_kernel.widgets_sagenb[0]>", line 1, in <module>
        from ipywidgets.widgets.tests.utils import setup_test_comm
      File "/sage/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/ipywidgets/widgets/tests/utils.py", line 50, in <module>
        orig_comm = ipykernel.comm.comm.BaseComm
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    AttributeError: module 'ipykernel.comm.comm' has no attribute 'BaseComm'
**********************************************************************
File "sage/repl/ipython_kernel/widgets_sagenb.py", line 12, in sage.repl.ipython_kernel.widgets_sagenb
Failed example:
    setup_test_comm()
Exception raised:
    Traceback (most recent call last):
      File "/sage/src/sage/doctest/forker.py", line 709, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/sage/src/sage/doctest/forker.py", line 1144, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.ipython_kernel.widgets_sagenb[1]>", line 1, in <module>
        setup_test_comm()
        ^^^^^^^^^^^^^^^
    NameError: name 'setup_test_comm' is not defined
**********************************************************************
1 item had failures:
   2 of   5 in sage.repl.ipython_kernel.widgets_sagenb

[egourgoulhon]

Thank you for this update!

Built successfully with system Python (3.10.12) on Ubuntu 22.04 after pulling this PR branch into a fresh git clone of sage/develop.

• got the same doctest errors as Notebook 7, ipykernel 6.27, ipython 8.17 #36129 (comment) above.
• ran the Jupyter notebook with sage -n successfully on the notebook https://github.com/egourgoulhon/SageMathTest/blob/master/Notebooks/test_display.ipynb : this tests various displays: LaTeX, interactive widgets, 2d and 3d plots, animated 3d plots, Matplotlib interface, etc., especially in the scope of %display latex. The Matplotlib interface (activated via %matplotlib widget) required the package ipympl, which had to be installed by ./sage -pip install ipympl. Could ipympl be made a standard package, so that the interactive Matplotlib interface works out of the box?
• when running the same notebook within Jupyterlab (via sage -n and a click on the the Jupyterlab button), the LaTeX displays turned out to be ugly, while they are fine with the Jupyter notebook interface.

[mkoeppe]

Thanks for testing!

[mkoeppe]

Could ipympl be made a standard package, so that the interactive Matplotlib interface works out of the box?

Done in 26a4773

[mkoeppe]

• when running the same notebook within Jupyterlab (via sage -n and a click on the the Jupyterlab button), the LaTeX displays turned out to be ugly, while they are fine with the Jupyter notebook interface.

Could you check if installing https://pypi.org/project/jupyterlab-mathjax3/ makes a difference?

[egourgoulhon]

• when running the same notebook within Jupyterlab (via sage -n and a click on the the Jupyterlab button), the LaTeX displays turned out to be ugly, while they are fine with the Jupyter notebook interface.

Could you check if installing https://pypi.org/project/jupyterlab-mathjax3/ makes a difference?

Actually, this turned out to be not necessary: if one launches Jupyterlab via sage -n jupyterlab the LaTeX displays are OK. They are only ugly (weird fonts) when one launches Jupyterlab by cliking on the JupyterLab button in the top right menu of a notebook opened in the classic notebook mode, i.e. launched with sage -n. Since sage -n jupyterlab is the standard way (as currently documented) to run Sage with Jupyterlab at this stage, the issue is thus quite minor.

[egourgoulhon]

Thank you for making ipympl standard! I've just checked it in a brand new install of this PR branch merged in Sage 10.2.beta0. This indeed makes the interactive Matplolib window work out of the box in Sage's Jupyterlab.

[orlitzky]

File "sage/repl/ipython_kernel/kernel.py", line 48, in sage.repl.ipython_kernel.kernel.SageKernel.__init__
Failed example:
    from sage.repl.ipython_kernel.kernel import SageKernel
Expected nothing
Got:
    0.00s - Debugger warning: It seems that frozen modules are being used, which may
    0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
    0.00s - to python to disable frozen modules.
    0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.

That's one of the failures I have from #36141

Adding PYDEVD_DISABLE_FILE_VALIDATION=1 to sage-env would be an easy solution unless someone needs to add breakpoints to a frozen python core module. It's not unthinkable, but the alternative to forbidding it (disabling frozen modules whenever we might import the sage kernel) seems worse.

What do you think? I can still push a fix to #36141.

[mkoeppe]

Is this coming from debugpy?

[orlitzky]

Is this coming from debugpy?

debugpy's bundled pydevd to be exact. The warning is raised when you import debugpy, like ipykernel does:

sage: from debugpy.server import api
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.

It's a new warning but the behavior hasn't actually changed. If you were already using python-3.11, you can't put breakpoints in frozen modules. Now you still can't, and you get a warning about it at an irrelevant time. Setting PYDEVD_DISABLE_FILE_VALIDATION=1 essentially puts you back where you were: no breakpoints no warning.

[egourgoulhon]

I've just noticed an issue with the Notebook 7, both with jupyterlab and the classic notebook: in %display latex mode, long outputs are truncated, with no scrolling bar:

while in Sage 10.1 (running the Notebook 6) the output is fully displayed on several lines:

[dimpase]

to me Rust sounds less insane than Node.js. Anyway, can we use binary wheels for a package which needs Rust to build from source?

[mkoeppe]

can we use binary wheels

so far we don't have a mechanism for this for standard packages

[dimpase]

can we use binary wheels

so far we don't have a mechanism for this for standard packages

This is something that looks very meaningful to have. Is it somewhere in a written roadmap?

a workaround for this would be to also allow packages installed by the "building" user, via
pip install ... --user, or use --enable-system-site-packages from #36256 in a venv (e.g. pyenv).

[mkoeppe]

We can of course install a package from binary wheels by demoting it to a "pip" package.
So far, by policy, standard packages cannot be "pip" packages because that requires an internet connection on install.

[mkoeppe]

This package (jsonschema) is almost only used on the Jupyter notebook side of things, so it would not run in the same Python as the Sage library. For such packages, we could decide to lower our standards regarding "installability from source". Then binary wheels (maybe implemented using an "ARCH" variable in checksums.ini etc.), or alternatively micromamba (#35585) could be OK.

(But jsonschema may be entangled with the Sage library via nbformat -> jupyter_sphinx; used for the interactive documentation build; see #36144.)

[dimpase]

We can of course install a package from binary wheels by demoting it to a "pip" package. So far, by policy, standard packages cannot be "pip" packages because that requires an internet connection on install.

Can't one pip-install packages from pre-fetched binary wheels?

[dimpase]

exceptiongroup is used in ipython docs, as well as in it itself (only for Python3 version < 3.11)
cf. https://github.com/ipython/ipython/blob/dd7f1e1894395f59649a84f94af8c9710202b08f/setup.cfg#L34

[mkoeppe]

Apparently this is another backport package that we need to add. exceptiongroup is a new feature in Python 3.11.
https://pypi.org/project/exceptiongroup/

[mkoeppe]

Tested successfully with ./configure --with-python=python3.9

[orlitzky]

Apparently this is another backport package that we need to add. exceptiongroup is a new feature in Python 3.11. https://pypi.org/project/exceptiongroup/

Can you add an spkg-configure.m4 to skip it for >=python-3.11 pleeeeeease? A copy/paste of #36776 should work.

[orlitzky]

I don't think we should mention it here unless we plan to make this a dependency of sagelib rather than something that just incidentally works?

[mkoeppe]

Hm... I agree that we should mention it there if and only if we make it a sagelib dependency.

I could go with either way -- adding such a backport package in general enables developers to use Python 3.11 features earlier, which is a good thing.

But I'll take out this commit here; we can consider it on another PR.

[mkoeppe]

Can you add an spkg-configure.m4 to skip it for >=python-3.11 pleeeeeease? A copy/paste of #36776 should work.

I'd rather not do it here. By the way, I think your solution in #36776 is not complete, as it seems to disregard the case --without-system-python3

[github-actions]

Documentation preview for this PR (built with commit 744e4b7; changes) is ready! 🎉

[egourgoulhon]

Thanks for having added the exceptiongroup package. Sage, as built with Ubuntu 22.04's default python (3.10.12), can now be launched without any error. I've tested both the (new Jupyterlab-based) Jupyter notebook and Jupyterlab on these two notebooks: test_display.ipynb and first_contact.ipynb.
Everything works nicely, except for two minor points in test_display.ipynb:

• in cell 18, the Ipython-native LaTeX display fails in the framework of %display latex
• in cell 21, which invokes %matplotlib widget, a deprecation warning shows up:

/home/eric/sage/10.2-test/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/traitlets/traitlets.py:1387: DeprecationWarning: Passing unrecognized arguments to super(Toolbar).__init__().
NavigationToolbar2WebAgg.__init__() missing 1 required positional argument: 'canvas'
This is deprecated in traitlets 4.2.This error will be raised in a future release of traitlets.

None of these issues is related to this PR, so I guess they can be fixed in a subsequent PR.

Side note: it is very nice to have Jupyterlab working out of the box, moreover in a recent version. Thank you Matthias!

[mkoeppe]

Thanks a lot for testing!

[egourgoulhon]

Apparently, with the new setting, the latex display does not work when there is no internet connection; see #36914.