Rework Voila-ipywidgets interface

[martinRenou]

cc. @SylvainCorlay @jtpio @trungleduc

Problem

The current state is:

• ipywidgets provides the jupyterlab-manager that is loaded dynamically by JupyterLab, meaning JupyterLab doesn't care if it's ipywidgets 7 or 8, it will load dynamically the manager and just work.
• voila provides the manager for ipywidgets 7, meaning it cannot work with ipywidgets 8 unless we update voila, but then voila would not work anymore with ipywidgets 7 unless we provide both the manager for 8 and 7 in the page (and if we are lucky that both don't collide).

Proposed Solution

My proposal is to:

• either rework the jupyterlab-manager in ipywidgets to make it work for free with voila so that voila loads it dynamically the same way JupyterLab does (goes in line with Switch to a lab-based app for the Voila frontend voila-dashboards/voila#846)
• either have two packages provided by ipywidgets: jupyterlab-manager and voila-manager

Currently I guess the jupyterlab-manager is too much related to the Notebook, but making it more general purpose could be good (so widgets could work with the jupyterlab notebook, voila, replite etc)

[jtpio]

• either rework the jupyterlab-manager in ipywidgets to make it work for free with voila so that voila loads it dynamically the same way JupyterLab does

That was the initial thought when starting voila-dashboards/voila#846. Since the lab-based Voila frontend will support prebuilt JupyterLab extensions, it could then pick up the prebuilt extension for ipywidget. But since the Voila frontend does not expose a Notebook object on the page via the INotebookTracker, the jupyterlab-manager plugin does not find the widgets to render.

Maybe Voila should still provide its own manager. The switch to a lab-based application will still be very breaking anyways so the update to ipywidgets 8 can make sense.

[jasongrout]

Some relevant thoughts are explored more in #3429.

I think most widget packages will support both ipywidgets 7 managers and ipywidgets 8 managers.

Another approach then would be to load the ipywidgets 8 jlab widget manager, and load both the ipywidgets 7 core widgets and the ipywidgets 8 core widgets. The widget manager should use the appropriate package when called for. However, one problem with that is the CSS will conflict, so #3429 (comment) proposes scoping the 7.0 core widget css separate from the 8.0 core widget css.

From #3429, it looks like VS Code is just determining which ipywidgets version to load initially. That would line up with your proposal of just loading whatever js was installed in the environment.

[trungleduc]

Maybe Voila should still provide its own manager. The switch to a lab-based application will still be very breaking anyways so the update to ipywidgets 8 can make sense.

👍 on this, I think it's reasonable to support only ipywidgets 8 in voila 0.4.0. We will keep backporting the non-breaking changes to the 0.3.x branch.