Pdoc Does Not Pick Up PyO3/pybind11 Submodules

[Stock84-dev]

Problem Description

Documentation for .pyi files is only generated for __init__.pyi. If a package includes other *.pyi files, then they are ignored.

Steps to reproduce the behavior:

1. pip install opendal==0.41
2. pdoc opendal
   opendal includes layers.pyi which is shown here, but the submodule isn't displayed in docs.

ls ~/.local/lib/python3.10/site-packages/opendal
__init__.py  __init__.pyi  layers.pyi  _opendal.abi3.so  __pycache__

System Information

pdoc: 14.1.0
Python: 3.10.8
Platform: Linux-6.1.31-2-MANJARO-x86_64-with-glibc2.38

[mhils]

Thanks! The problem is not so much the pyi file, but the fact that we don't pick up the submodule:

$ pdoc opendal.layers -o .render
Warn: Cannot find spec for opendal.layers (from opendal.layers):
Traceback (most recent call last):
  File "/Users/user/git/pdoc/pdoc/extract.py", line 62, in walk_specs
    raise ModuleNotFoundError(modname)
ModuleNotFoundError: opendal.layers
 (/Users/user/git/pdoc/pdoc/__init__.py:506)
Traceback (most recent call last):
  File "/Users/user/git/pdoc/venv/bin/pdoc", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/Users/user/git/pdoc/pdoc/__main__.py", line 199, in cli
    pdoc.pdoc(
  File "/Users/user/git/pdoc/pdoc/__init__.py", line 506, in pdoc
    for module_name in extract.walk_specs(modules):
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/git/pdoc/pdoc/extract.py", line 85, in walk_specs
    raise ValueError(
ValueError: No modules found matching spec: opendal.layers

$ pdoc opendal opendal.layers -o .render
Warn: Cannot find spec for opendal.layers (from opendal.layers):
Traceback (most recent call last):
  File "/Users/user/git/pdoc/pdoc/extract.py", line 60, in walk_specs
    modspec = importlib.util.find_spec(modname)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib.util>", line 114, in find_spec
ValueError: opendal.layers.__spec__ is None
 (/Users/user/git/pdoc/pdoc/__init__.py:506)

Gotta figure out how we solve this one. :)

[Stock84-dev]

They use a workaround to allow importing a submodule directly which might be relevant:

• ModuleNotFoundError: No module named 'supermodule.submodule'; 'supermodule' is not a package PyO3/pyo3#759
• https://github.com/apache/incubator-opendal/blob/c52061a3682b55bdcc5724481b207666db9b0494/bindings/python/src/lib.rs#L577

[evguran]

I've got the similar issue with empty doc.submodules instance in case of builtin modules created via pybind11. the pkgutils couldn't find it, because when importing submodules, no matter via importlib or import there were no __path__, __spec__, __package__ and other attributes added to the module, although normally it should be.
I've solved the issue by adding the recursive code to walk through the module, looks its dir(module), and then walk through the items checking inspect.is_module(m).
Then I've manually added docs to the doc.submodule of each tree node.
Hope this helps .

[mhils]

This is super helpful, thanks @evguran! 🍰

[mhils]

Minimal repro: https://github.com/mitmproxy/pdoc-pyo3-sample-library

[mhils]

I have a candidate PR to fix this over at #643. Could you folks test that and report back if it fixes your issues? 😃

pip install git+https://github.com/mhils/pdoc@pyo3-submodules

(cc @MarquessV who may not be subscribed but linked this issue a few days ago)