Converting from sphinx-nbextensions to nbsphinx

[Jacob-Stevens-Haas]

Python: 3.8.10
nbsphinx: 0.8.9

Hi, I'm trying convert from sphinx-nbextensions to nbsphinx since the former is deprecated in favor of this repo. I've left the title general so as to avoid giving you an XY problem, but my specific issue is that sphinx-nbextensions finds notebooks & a README.rst in the project directory (using example_gallery_config in conf.py) and creates docs/examples/, adding the notebook .rst files and an index.rst. This is somewhat analogous to how apidoc_module_dir takes files from package/ and puts them in docs/api. My project layout is

source_tree/
\
 |-package/
 |-examples/ #has some subdirectories with example.py and example.ipynb
 |-docs/

My index.rst has the toctrees:

.. toctree::
   :maxdepth: 1
   :caption: User Guide

   API Documentation <api/pysindy>
   Examples <examples/index>
   Practical tips <tips>

.. toctree::
   :maxdepth: 1
   :caption: Useful links

I'm not sure how to configure nbsphinx to work. Just swapping out the extension one-for-one gives the:
toctree contains reference to nonexisting document 'examples/index' error because nbsphinx is not looking in source_tree/examples/. I'd like to be able to tell it to do that. I don't see any analogous config option to apidoc_module_dir or example_gallery_config.

[Jacob-Stevens-Haas]

I can move the examples manually into docs - and reconfigure any tools that also manage the notebooks - but then I get the sphinx error that my .ipynb files aren't included in any toctree, which means I can't guard PRs with a -W build. The solution here suggests adding these files to exclude_patterns, but then nbsphinx excludes these as well.

I suppose another hacky way to solve this is to write a setup(app) function in conf.py that just copies the examples from examples/ to docs/examples/, but unless I can down that structure, I'll still get the toctree errors.

[mgeier]

I can move the examples manually into docs

With nbsphinx, Jupyter notebooks are used as Sphinx source files, so yes, they have to be in the source directory (or in a subdirectory thereof).

If you don't want that, you can try https://github.com/vidartf/nbsphinx-link, but this has its own limitations.

You can also try to use symbolic links, but this might also lead to problems, see #49.

but then I get the sphinx error that my .ipynb files aren't included in any toctree

Sphinx expects all source files to be in a toctree and normally that's what you want, so you should put them into a toctree.

If you have a reason not to put them into a toctree, you can mark them as "orphan" (see https://nbsphinx.readthedocs.io/en/0.8.9/orphan.html), which gets rid of the warning.

[Jacob-Stevens-Haas]

Thanks @mgeier! Reading further on sphinx's page, I can apparently treat conf.py like an extension. I can try defining setup(app) to copy/move the notebooks into docs... But would it really be that simple? I could also maybe register a high-priority (low int) callback on build-finished to move them back/delete copies if they're not in a toctree, but not sure whether the toctree error is emitted before or after that event. Sorry if these questions are a bit basic, as I've never really used event-driven programming.

so you should put them into a toctree

Good point. If my setup(app) also moves examples/README.rst to docs/examples/index.rst, would that clear the error for all documents in all subdirectories of docs/examples?

[mgeier]

I can apparently treat conf.py like an extension. I can try defining setup(app) to copy/move the notebooks into docs...

Yes, but that seems overkill to me. I would only use that as a last resort, if nothing else works.

I don't really know how the structure of your project looks like, so I can't really give any recommendations.

I think it would be helpful if you could create and share a new minimal dummy project that has the same structure as you would like to have in your real project. Then we can experiment with that.

[Jacob-Stevens-Haas]

I don't really know how the structure of your project looks like, so I can't really give any recommendations.

I think it would be helpful if you could create and share a new minimal dummy project that has the same structure as you would like to have in your real project. Then we can experiment with that.

Larger, real repo is pysindy, but I built a simpler MWE for this issue. I added the conf.py edits that i mentioned, so this project's docs build "correctly" and without toctree errors. If you want to close this issue as completed, it's fine with me - other people with the same issue will be able to find this as a how-to guide.

More broadly, the problem is that there are other tools for dealing with notebooks in a repo. Any tool that doesn't implement globs/regexs for configuring where to find files has to assume something about directory structure - which may conflict with other tools that do the same. Juptext has it's own issues in that regard, though I can't immediately think of a conflict with nbsphinx. But my test infrastructure does conflict: I need notebooks to offload their data generation to local modules in order to do import substitution. Those local modules aren't source files for sphinx. If they're colocated in docs/examples/, I would have to add them to a toctree in order to silence errors.

[mgeier]

Thanks for the minimal(-ish) example!

This is still quite a lot to digest ...

Is the apidoc stuff relevant to the question at hand?

In general, I think generating temporary files in the source directory should be avoided but that might not always be possible, see sphinx-doc/sphinx#10641

I also don't understand why you have both .ipynb and the matching .py in the repo ... why is that?
I would try to keep only one of them and generate the other one on demand.
If you only keep the .py, nbsphinx will automatically generate a .ipynb for you.

Once those things are clarified (and ideally simplified), I would like to come back to your original question.

Those local modules aren't source files for sphinx. If they're colocated in docs/examples/, I would have to add them to a toctree in order to silence errors.

I don't understand. Can you please give an example for when those errors happen?

[mgeier]

I'm closing this for lack of response, but if there is still something to discuss, feel free to leave another comment!