feat: Allow the use of *-stubs packages

[romain-intel]

Small change to look for X-stubs first when looking for the X module. This better respects PEP 561.

Incidental fix to properly recognize a stubs only package as a package (and not a namespace package).

Please let me know if this is the correct approach (it works for my use case). I can also try to add tests if this is something that is mergeable. I tried to keep my changes to a minimal (being totally unfamiliar with the code).

[pawamoy]

Hello, thanks for the PR! Could you describe your use-case?

A few questions regarding stubs-only packages:

• Can you document objects exposed in __init__ modules with a stubs-only package?
• Can you document instance-only attributes (defined in __init__ methods) with a stubs-only package?

It's true that stubs packages take precedence over regular packages, but this is for type checkers, and Griffe is not a type checker.

A few other remarks:

• The code in this PR does not seem to handle stubs namespace packages.
• The added try/except block will double (or likely much more) the time it takes to find a regular package (which is ~99% of the cases).

Based on this, I wonder if we shouldn't add a load_stubs_package parameter somewhere, to find, load and merge the stubs-only package data into the regular package data.

[romain-intel]

Hey @pawamoy -- thanks for your quick response.

My use case is that I help develop Metaflow (see https://github.com/Netflix/metaflow) and we are considering releasing a stubs package for Metaflow because, for various reasons, Metaflow creates "dynamic" functions of sorts (I can get into details if you want them but here is the PR Netflix/metaflow#1557 and the README (here: https://github.com/Netflix/metaflow/pull/1557/files#diff-5405972c6b3d9df037790c75db2dc8fb7c1987c8707ae4ad84c39adcb8da00a0) should kind of explain what it is trying to do (PR is not yet fully ready so still has a few rough edges).

We then have a docsite that uses Griffe and I wanted to do the following:

• install the stubs only package (let's call it metaflow-stubs) in the Griffe environment (or wherever we are rendering our doc website)
• use ::: metaflow.batch for example. This is not an actual method that exists in the package but it does exist in that generated stubs package.

Note that one workaround is I could do ::: metaflow-stubs.batch (with that tiny change to consider __init__.pyi as a proper file (else it finds the package but skips the __init__.pyi which is, in my case, the most relevant file :).

I am not sure I understand your two questions but I believe you can document anything you want in stubs packages. The only thing I could think of that would probably not make sense are types of variables inside methods/functions for example. In terms of documenting classes and their methods (including init), yes, you can definitely do that. Basically, the stubs only package contains just the .pyi files which all contain methods that are all ... but all other annotations and docstrings can be kept.

Yes, I didn't test it for stubs namespace packages. I also didn't handle partial stub packages (https://peps.python.org/pep-0561/#partial-stub-packages). I believe those two cases would probably be a little more complicated because technically they are incomplete so I suppose (not 100% sure) that the information from the stubs packages would have to be "merged" with that of the actual package. With stubs packages (a complete one), you don't need to look "further down" so to speak and you can stop at the stubs package.

It's true that there is a cost to look for the uncommon case (not a fan of that). I didn't think it would be so significant but I don't know the code as well. It will check a few locations and if the package does not exist, it should just return out pretty quickly (hopefully). I thought what took time was actually traversing the packages but there there shouldn't be much to traverse. A flag might make sense though (an option that could be passed by the user to say: "please look for a stubs package".

Let me know if this makes sense and/or if you have more questions. Happy to help. I would say, as well, that for my use case, I think I could do away with just the small change of considering init.pyi the same as init.py and I could just rename everything X-stubs. I haven't tried it out but I suspect it would work. If I am the only weird one in this situation, it may be the easiest :).

[pawamoy]

Thanks for the detailed explanation!

What I meant by

Can you document objects exposed in init modules with a stubs-only package?

is that if you mirror your package structure in the stubs, you can't document object imported from submodules and exposed in __init__ modules. But I guess that you don't mirror the entire structure, and document every public object in the top-level __init__ module. Or you mirror just the public facing modules.

And by

Can you document instance-only attributes (defined in init methods) with a stubs-only package?

I meant, how do you document instance_var below in a stubs-only package?

class A:
    class_var: int = 0

    def __init__(self):
        self.instance_var: int = 0

I think it's not possible 🤔 Unless you document it as a class variable, which wouldn't be so bad.

---

As tempting as ::: metaflow-stubs.batch looks, I won't let you do that, because your inventory would then wrongly use the stubs package name in the object paths/identifiers, preventing anyone from cross-linking your docs 😅

Anyway. I'm not opposed to the idea of allowing users to load data only from stubs packages. I'd just prefer that we avoid always looking for a stubs package before the regular package. Or maybe we could always try to find both: find package, then find stubs, if package not found return stubs, if package found return package+stubs (will be merged). Do you plan on installing only the stubs package in the Griffe environment, or the actual package as well?

[romain-intel]

To answer the questions:

• you can document whatever you want in __init__.pyi. In my case for example, if something is imported, I choose to document it (based on the way I did the stubs generation) but you could also not have it documented there and just document it in the submodule. So for example, if in your mypackage init.py you had:

from .foo import bar

where bar is a function, you could technically add a:

def bar(...):
  ...

in __init__.pyi and that would work but you could also choose to document bar directly in foo.py and that would also work.

• you are correct that you don't document any variables that are set in init.pyi (at least I haven't seen that use case but I am not a huge expert either).
• In terms of installing stubs and regular package. I can do both. I think I technically only need the stubs in terms of what I am thinking of documenting but I can see a case where you may need both.

I get your concern about looking for two packages. One option, which you suggested earlier, is to also have a flag that users can set that instructs griffe to look for stubs packages so that if users don't need that functionality, they don't pay for it at all.

We could definitely do the merging too. I think you already have a function that merges .pyi and .py files from within the same package so this would be somewhat identical just that the files would be the "same files" with different root directories (ie: be in different packages). I can attempt that if that is something you think would be worthwhile. In short, happy to put in the work to make this PR more "production ready" and better (although I may need help on things like "add a configuration option" which I haven't looked for too much yet).

[pawamoy]

I've pushed changes to your branch, I hope you don't mind!

With these changes I think we now also support namespace packages for stubs. I added tests for the finder, and we need more tests for the loader, to check that regular/namespace package+stubs combo are correctly loaded and merged together.

[romain-intel]

I don't mind at all!!! Thanks for taking the time. Much appreciated. I'll try to add tests to cover those other use cases.

[pawamoy]

I should be able to add these tests tomorrow, don't worry about it :)

[romain-intel]

You make "contributing" (generous characterization) to your repo very nice!

[pawamoy]

Added some tests, that helped discover an oversight and bug in the stubs merger. I also added a CLI option. Do you use Griffe directly or through mkdocstrings?

[romain-intel]

Awesome. Let me try it out this weekend. I use it through mkdocstrings.

[romain-intel]

I apologize for the delay -- it's been a busy past few days. I plan on trying this out tomorrow (ie: this branch in my setup). Apologies again.

[pawamoy]

No need for apologies! We're all busy 🙂 Thank you for following up and testing out the branch!

[romain-intel]

I tested with -B and it seems to work fine. One comment though is that, in some cases, the merge of the .py and .pyi fails because of a type mismatch. I have several cases of this but one is TYPE_CHECKING which is an attribute in the pyi file but an alias in the .py file. In this case, I think the merge should "favor" the .pyi file.

Next step for me is getting it to work with mkdocstrings but that shouldn't be too bad hopefully.

[pawamoy]

Thanks for reporting back. Do you mean that you get debug log messages when the objects have different kinds, or does Griffe actually crash somehow? Do you have other examples of objects not being merged because they have different kinds? TYPE_CHECKING is probably not important at all for API docs.

[pawamoy]

The type mismatch you mention should only happen when one of the two objects is an alias and cannot be resolved. If it cannot be resolved, it probably means that it is imported from another package (like TYPE_CHECKING coming from typing). In that case, this other package should be pre-loaded. But I can imagine that in some cases we'd want to favor the stubs over the regular member when there's a type mismatch. I'd need some examples though, to make sure 🙂 Anyway, this can and should be done in another PR! I'm going to merge this so we can release it in the next version.