Integrating nbmake with jupyter-book

[alex-treebeard]

Hey folks,

I mentioned this to @choldgraf in an email but thought it's best to make the discussion transparent.

I'm working on a new library for notebook testing which will allow me to share a load of functionality tangled in treebeard in a faster, more open way.

The idea with nbmake is to build ipynbs into jupyterbooks via pytest.

Why pytest? Mostly because of parallelisation features in xdist, but also due to the massive ecosystem of other plugins and IDEs which can be adapted to work well with notebooks through this approach.

With pytest as a front end for this capability I consider jupyter-book to be the best runtime/package format compared to alternatives such as papermill and nbclient because the output is a deployable site and the directives support is so extensive.

This should help users of notebooks in CI pipelines (of which I have spoken to at universities/companies)

Question

It seems like Jupyter Book is too new to have a robust public API so I am using adapters like so, and as a result I want to de-risk my approach a little:

Please tell me what type of issues I may run into doing the following:

1. When testing a book, each ipynb is separately executed using jb build
2. After each single-page book is built, the jupyter_execute outputs are combined into a temp-dir, which looks like the original repo
3. Build the final book using existing _toc.yml, if appropriate.

I'm specifically concerned about links between pages being broken due to separate execution.

Finally, please let me know if you think this should be done fundamentally differently. E.g. putting jb in front of pytest.

Thanks!

E: I'm also reading through MyST-NB as it may be cleaner to have the pytest plugin write to the jupyter-cache.

[chrisjsewell]

Thanks @alex-treebeard its definitely the kind of thing I want to look into.
But.. I'm not sure if pytest is definitely the right route, and this should not just be part of jupyter-cache, or how the pytest/jupyter-cache integration would work.
(Not to say that I'm ruling it out.)

Just for notebook testing, I'm also not sure what the benefits of including jupyter-book are? is the goal to also test the Markdown?

FYI, also of relevance may be another one of my packages: https://github.com/chrisjsewell/pytest-notebook/

[alex-treebeard]

Hey Chris thanks for the heads up on pytest-notebook. I can see our interests overlap.

I'm far from convinced I have the right approach universally for notebook usage. My goal is actually just to deprecate a closed source notebook viewing tool that I am finding unmaintainable.

I should be getting a release of nbmake out before xmas and I'm sure it will be an interesting case study going forward for others interested in nb testing.

benefits of including jupyter-book

Observability -- if you wait an hour for 30 notebooks to run and some fail, ideally you can see exactly what happened.

I'm going to close this as I've moved forward roughly as described, but expect to be contributing issues/fixes to some of the execution-layer repos in the coming weeks/months.

Thanks

[chrisjsewell]

Look forward to it 😀