Generate documentation and stub file for nanobind Python module

[yosh-matsuda]

I've been trying to use a Python module built with nanobind to automatically generate documentation and stub files. My goal is to achieve this using Sphinx and mypy-stubgen. This has now been almost achieved for my use case with my own small fixes:

• mypy-nanobind
• sphinx-nanobind

Here, I would like to share what nanobind can do to make them possible without such modifications in my observations. If the following problems are solved, Sphinx and mypy-stubgen would work almost perfectly I think.

1. nanobind.nb_func

The nanobind bound functions are not recognized as Python functions (inspect.isfunction) because the type of the boud functions seems to be nanobind.nb_func defined in nb_internal.h. My question is, is it possible to make the boud function as if it is built-in like pybind11? I don't know how to deal with this as my understanding of nanobind internals and cpython is limited.

2. Incompatible type hints

I've noticed that the stubgen cannot recognize type names in some cases. Can I ask if there is anything we should fix in nanobind?

• The mypy-stubgen cannot handle <anonymous>(...) -> ..., on the other hand, it supports pybind11-style anonymous type hints like (...) -> ... implemented in Improve stubgenc type detection on PyBind11 properties python/mypy#8995. The nanobind may not need <anonymous> in the docstring if compatibility with pybind11 is important, otherwise, I'm afraid that this should be handled on the mypy side.
• The ndarray type annotations, (e.g. numpy.ndarray[dtype=float32, shape=(2, 4)]) cannot be parsed as type in the stubgen. This is probably due to the = ( ) characters, which are not normally used in type names. I have changed the ndarray type name generation in ndarray.h and confirmed that numpy.ndarray[dtype[float32], shape[2, 4]] works well for example.

[wjakob]

Regarding nanobind.nb_func: this is a required part of nanobind (and responsible for some of the speedups). It may be necessary to hot-patch inspect.isfunction (or ask the maintainers to consider a more general form that recognizes other things as functions as well).

I am open to removing the <anonymous> string. The tensor type annotations are a tricky one and perhaps a good topic for a larger discussion. For example, the shape is a tuple, and it does not make sense to me to pass it via [ .. ].

[wjakob]

Ultimately, it may be unrealistic for nanobind to generate type annotations that are well-suited to each tool. But what can likely be done is to create a custom stub generator with some kind of callback/cleanup hooks that can be used to introduce transformation rules.

[yosh-matsuda]

Thanks for your reply, I understand that nb_func object is required for nanobind.

I am open to removing the string.

I will create a pull request to remove the <anonymous> string later.

The tensor type annotations are a tricky one and perhaps a good topic for a larger discussion. For example, the shape is a tuple, and it does not make sense to me to pass it via [ .. ].

The issue of how to annotate an attribute other than dtype, e.g. shape, in Numpy may be helpful. If nanobind strictly provided ndarray type hits, I think we can use typing.Annotated to add metadata, which may be useful.

[wjakob]

I am not a fan of typing.Annotated : it's not very readable on the user's end (if shown in error messages) and furthermore not used for type checking. Basically we are taking all the extra type information that MyPy cannot deal with and then sort of throwing it into the trash ;).

My suggestion would be that we leave nanobind type annotations the way that they are but then think about how a customized stub generator could project the signatures onto something that MyPy can deal with (i.e. additional potentially lossy postpossessing)