PEP561: Clarify stub-only namespace package behavior

[nipunn1313]

Clarify requirements on authors of stub-only packages within namespace packages, as well as requirements for type checkers handling such packages.

Additional context in
microsoft/pyright#2113
python/typeshed#5800

[the-knights-who-say-ni]

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA).

Recognized GitHub username

We couldn't find a bugs.python.org (b.p.o) account corresponding to the following GitHub usernames:

@nipunn1313

This might be simply due to a missing "GitHub Name" entry in one's b.p.o account settings. This is necessary for legal reasons before we can look at this contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

You can check yourself to see if the CLA has been received.

Thanks again for the contribution, we look forward to reviewing it!

[gvanrossum]

I like this. Since this is a PEP you're proposing to change, let me ping the author, @ethanhs before I merge your changes. (I presume you've filled out the CLA? It takes at least a business day, usually.)

[nipunn1313]

I just filled out the CLA - it will take a day or two to propagate.

[nipunn1313]

Yep - I think @ethanhs should definitely have eyes on this before it goes any further. ethans is aware from python/typeshed#5800 (comment)

[nipunn1313]

This clarification was inspired by googleapis/python-firestore#447 - around the module google/cloud/firestore.py which exists within a package distribution in a way that can't be py.typed

[emmatyping]

I don't think is quite right. In the above issue, the py.typed file belongs inside the google/cloud folder, as google is the namespace package (unless I am mistaken), so the py.typed file lives in each sub-package of the namespace package.

[nipunn1313]

google.cloud is also a namespace package (sorry for confusion)

[nipunn1313]

It's a nested namespace package

[emmatyping]

I believe you should still list the sub-package of the outermost package as typed (even if partial).

[nipunn1313]

Just to make sure we're on the same page - I don't think the preexisting PEP unambiguously specifies this case (single-file modules in possibly nested namespace packages) - we are discussing/reasoning about what makes the most sense to make as the specification here.

The sub-namespace-package (google.cloud in this case) will be populated by several different pip distributions (https://pypi.org/search/?q=google-cloud-). I don't think it works - or makes sense - for each of those pip distributions to place a py.typed partial in google.cloud.

As a concrete example - for something like google/cloud/firestore.py which is fully typed - It doesn't make sense to have a google/cloud/py.typed indicating full typed-ness - since google/cloud could have other entries from other distributions. Rather we must require it to be google/cloud/firestore/__init__.py with google/cloud/firestore/py.typed so that typecheckers can know that firestore is fully typed.

[emmatyping]

Yeah that seems reasonable.

[henribru]

@nipunn1313 I ran into something related to this while working on typing for https://pypi.org/project/googleapis-common-protos/ (using the excellent mypy-protobuf, thanks for your work!) and was wondering if you could confirm if my understanding of the updated spec is correct.

Like most Google distributions this distribution contains a namespace google package, which contains multiple subpackages. Here's the tree showing one particular subpackage:

google
├── __init__.py
├── rpc
│   ├── code_pb2.py
│   ├── code.proto
│   ├── context
│   │   ├── attribute_context_pb2.py
│   │   ├── attribute_context.proto
│   │   └── __init__.py
│   ├── error_details_pb2.py
│   ├── error_details.proto
│   ├── __init__.py
│   ├── README.md
│   ├── status_pb2.py
│   └── status.proto

Here both google and rpc are namespace packages (using declare_namespace/extend_path), while context is a regular package. As you can see google.rpc actually contains some modules. Presumably some other Google distribution could put other modules in this namespace. If I understand this discussion correctly, there's no way to distribute typing information for code_pb2.py, error_details_pb2.py and status_pb2.py and for this to work Google would have to refactor these into packages? Is the correct approach when making a stub-only package for this to simply leave them out, like this?

google-stubs
├── rpc
│   └── context
│       ├── attribute_context_pb2.pyi
│       ├── __init__.pyi
|       └── py.typed

[nipunn1313]

I actually wrote this spec modification with a case in mind - single-file-modules where types are included in the original package (googleapis-common-protos in your eg) - because in that case - a py.typed is required.

For a stub-only package - By the spec clarification - all namespace packages within stub-packages are to be considered partially typed by type checkers.

Type checkers should treat namespace packages within stub-packages as
incomplete since multiple distributions may populate them.
Regular packages within namespace packages in stub-package distributions
are considered complete unless a ``py.typed`` with ``partial\n`` is included.

Thus I think you can opt to do

google-stubs
├── rpc
│   └── context
│       ├── attribute_context_pb2.pyi
│       ├── __init__.pyi
|       └── py.typed

You may also add stubs for the remaining files (eg status_pb2.pyi), but be sure to omit the __init__.pyi for namespace packages. Typechecker should consider google-stubs and google-stubs/rpc as incomplete because they are namespace packages.

I haven't tested whether mypy/pyright obey this spec clarification yet - but I believe pyright recently made a change to make this work. I believe in my past testing mypy needed the --namespace-packages flag to treat these properly, but I didn't test this specific case.

[henribru]

Thank you for the very quick clarification! That's my bad for not properly distinguishing between typed "real" packages and stub-only packages. I guess the restriction on single-file modules is Google's problem if they ever decide to type this in the package itself. I'll just do this for my stub-only package then:

google-stubs
├── rpc
│   ├── code_pb2.pyi
│   ├── context
│   │   ├── attribute_context_pb2.pyi
│   │   ├── __init__.pyi
│   │   └── py.typed
│   ├── error_details_pb2.pyi
│   └── status_pb2.pyi

[emmatyping]

Looks pretty good, just a couple of small changes. Thanks for making these changes!

[emmatyping]

I don't think is quite right. In the above issue, the py.typed file belongs inside the google/cloud folder, as google is the namespace package (unless I am mistaken), so the py.typed file lives in each sub-package of the namespace package.

[nipunn1313]

Hi - ping to @ethanhs - just wanted to make sure this didn't get lost.

[emmatyping]

Thanks for working on this, and sorry for the delay in review!