What is the best practice for testing notebooks/markdown in a JupyterBook?

[robfalck]

I need to test the code in our documentation as part of our CI process. I want to verify that no changes to our package break examples in the documentation, and if they do then the documentation is changed accordingly.

I've tried adding cells at the end of notebooks that run assertions to test outputs. But when these assertions fail the build process still completes, resulting in incorrect documentation.

Is there currently a best practice for implementing this sort of functionality?

I can envision some sort of test tag for markdown code cells and notebooks that would result in the build process failing if exceptions are raised in that cell. This would make the JupyterBook build procedure a self-testing process for our CI, and help to ensure that our docs are always compatible with our software. I'd be happy to submit a feature request if others think this is a reasonable approach.