New project item: Jupyter Notebook

[spine-o-bot]

As discussed in the Spine Engine meeting, it could be interesting to create a new project item to embed a Jupyter Notebook that could be used both for data cleaning or for data visualisation.
The notebook embedded as project item can be a parametric notebook. A parametric notebook can be executed as a node and it accepts datastreams/parameters as input and outputs data as output parameters.
A Jupyter Notebook can be parameterised using the package called Papermill (https://papermill.readthedocs.io/en/latest/). The input parameters are the output of the previous node, while the output parameters will be the input to the following node.
Implementing such a project item can increase the flexibility of the spine workflow, allow users to create their own visualisations/data cleaning pipeline and can also open Spine to more case studies.
Parametric notebooks are supported in Dagster using a feature called Dagstermill, so the workflow can be executed by the spine engine without any problem (https://docs.dagster.io/docs/apidocs/libraries/dagstermill).
Let me know what you think.

[ptsavol]

Hi @RichieBrady. Can you merge the latest master into the notebook branch and make a pull request. Easier to see the changes that way. I got it working by installing the specific versions of spinetoolbox, spinedb_api, and spine_engine that the spine_item notebook branch now requires but it's very awkward. Also, there's this new dependency papermill. I think this should be added as a dependency of spine_engine by adding it to the install_requires list in spine_engine/setup.py. Also also, when you drag a blank notebook item to Design View, this happens

2021-04-09 13:18:40 ERROR: Category name Notebooks not found in ['Data Stores', 'Data Connections', 'Tools', 'Views', 'Importers', 'Exporters', 'Manipulators']
Traceback (most recent call last):
  File "C:\Users\ttepsa\OneDrive - Teknologian Tutkimuskeskus VTT\Documents\GIT\SPINETOOLBOX\spinetoolbox\project_commands.py", line 151, in redo
    category_ind, *project_tree_items, set_selected=self.set_selected, verbosity=self.verbosity
  File "C:\Users\ttepsa\OneDrive - Teknologian Tutkimuskeskus VTT\Documents\GIT\SPINETOOLBOX\spinetoolbox\project.py", line 267, in do_add_project_tree_items
    self._project_item_model.insert_item(project_tree_item, category_ind)
  File "C:\Users\ttepsa\OneDrive - Teknologian Tutkimuskeskus VTT\Documents\GIT\SPINETOOLBOX\spinetoolbox\mvcmodels\project_item_model.py", line 222, in insert_item
    parent_item = self.item(parent)
  File "C:\Users\ttepsa\OneDrive - Teknologian Tutkimuskeskus VTT\Documents\GIT\SPINETOOLBOX\spinetoolbox\mvcmodels\project_item_model.py", line 141, in item
    if not index.isValid():
AttributeError: 'NoneType' object has no attribute 'isValid'

But I suggest to do the merge first and see if this disappears.

[RichieBrady]

Hi @PekkaSavolainen I just pushed an update to the notebook_executor branch which has fixed bugs related to updating to the latest toolbox, engine, and spine_items. I will make a pull request later today and get the ball rolling with the merge.

The error you are getting is related to the toolbox which requires a small update to category,py and ui_main.py I will make a pull request for that as well.