🧪 Publish MyPy type coverage to Codecov

[webknjaz]

This patch also includes running MyPy against both Python 3.9 and 3.13 (3.13 has MyPy violations, so let's mark it out of the scope for now).

Quirks: https://discuss.python.org/t/is-there-any-tool-that-can-report-type-coverage-of-a-project/34962/4

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.82%. Comparing base (8a3307f) to head (09901bb).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3162      +/-   ##
==========================================
- Coverage   99.62%   98.82%   -0.81%     
==========================================
  Files         124      140      +16     
  Lines       18381    23041    +4660     
  Branches     1226     2282    +1056     
==========================================
+ Hits        18312    22770    +4458     
- Misses         47      149     +102     
- Partials       22      122     +100     

see 121 files with indirect coverage changes

[jakkdl]

mypy 3.13 is passing on main, no? https://github.com/python-trio/trio/actions/runs/12425306606/job/34691878825
So why would it fail here?

[webknjaz]

mypy 3.13 is passing on main, no? python-trio/trio/actions/runs/12425306606/job/34691878825
So why would it fail here?

@jakkdl this confusion is coming from not using an explicit --python-version. main actually runs checks against 3.9: https://github.com/python-trio/trio/blob/b43383d/pyproject.toml#L173C18-L173C23.

In other projects, I also put the lowest supported version in the config: https://github.com/aio-libs/yarl/blob/c1f6ef6/.mypy.ini#L2C18-L2C21. And I include as many settings as I can there, everything that is common.

But then, I also run MyPy against every other version in the range: https://github.com/aio-libs/yarl/blob/c1f6ef6/.pre-commit-config.yaml#L109-L187.

Ideally, it should run against every Python. Period. But it's expensive, so I either do every other, or min+max.

---

For history, this is how it's failing:

  src/trio/_path.py:246:22:246:58: error: Incompatible types in assignment (expression has type "Callable[[Path, str | PathLike[str], DefaultNamedArg(bool | None, 'case_sensitive')], Awaitable[bool]]", base class "PurePath" defined the type as "Callable[[PurePath, str | PathLike[str], DefaultNamedArg(bool | None, 'case_sensitive')], bool]")  [assignment]

(https://github.com/python-trio/trio/actions/runs/12425469238/job/34692341320?pr=3162#step:5:1083)

[webknjaz]

@jakkdl this should be ready now. Don't worry about Codecov's misleading coverage drops. Their PR diff views can't handle flag filtering well and sometimes display nonsense (requiring one to got to the HEAD commit page to view actual data).

[jakkdl]

[written before your latest comment]

Huh, this is really fascinating - I've never heard of type coverage. Though I'm not sure if the upsides of doing it this way outweigh the bugs/quirks when codecov tries to combine the reports.
Adding >100 # pragma: no cover markers for code that is in fact covered is quite a lot, and in case they later on do become uncovered then we won't get a warning.

I think our approach with pyright --verifytypes, where we commit a json with exceptions to the repo has worked quite well, and CI checks for changes so it's easy to notice if somebody tries to add an object without types/docstrings.

We could perhaps do something similar with the output from mypy, but even then I'm unsure if MyPy type coverage actually adds much of value to this repo - given that we both run --verifytypes and have a very strict mypy config. Before we'd completed transitioning to fully typed it probably would've been more useful, but at this point I'd be fairly confident in saying every single remaining Any has been checked multiple times if there's any reasonable way it could be replaced.

mypy 3.13 is passing on main, no? python-trio/trio/actions/runs/12425306606/job/34691878825
So why would it fail here?

@jakkdl this confusion is coming from not using an explicit --python-version. main actually runs checks against 3.9: https://github.com/python-trio/trio/blob/b43383d/pyproject.toml#L173C18-L173C23.

oh yeah derp. Yeah current setup prioritizes only having to specify the version in a single place.

Checking multiple mypy versions would be nice, but yeah it quickly becomes very slow when we also check three platforms.

  src/trio/_path.py:246:22:246:58: error: Incompatible types in assignment (expression has type "Callable[[Path, str | PathLike[str], DefaultNamedArg(bool | None, 'case_sensitive')], Awaitable[bool]]", base class "PurePath" defined the type as "Callable[[PurePath, str | PathLike[str], DefaultNamedArg(bool | None, 'case_sensitive')], bool]")  [assignment]

Looks like Path vs PurePath. That error is a pretty good argument for checking more than just 3.9 in CI, given that the method in question is guarded by if sys.version_info >= (3, 13):

[webknjaz]

@jakkdl for me, the benefit is getting visual representation of the type preciseness. MyPy can also output txt and HTML reports. The latter are useful locally, while the former could go into job summaries.

Here's an example of posting all those text outputs into job summaries: https://github.com/ansible/awx-plugins/actions/runs/12415268940#summary-34661329642.

Also, regarding the separation of pytest vs. MyPy coverage, it does compute distinct checks correctly. So running it allows visually seeing the percentages in PR statuses and also, setting some as required, but not the others.

In general, it's important to have the tests at 100% (the runtime/pytest check) so that we know that there's no dead code in tests. The next priority is 100% of the actual library code (also pytest/runtime) — for this one, it makes sense to use "no cover" pragmas. As for MyPy, I've seen cases where it's impossible to reach 100% precision with the current state of typing. But still, it's useful to see some coverage representation.

[jakkdl]

hmm, I'm not seeing a codecov check now. Is this a silent fail from having after_n_builds set too high?

[webknjaz]

I think Codecov might've been drunk at the time. The checks are in place now.

[A5rocks]

"type preciseness" is just tracking Anys, correct? I'm pretty sure we handle those explicitly with type ignore comments, having enabled plenty of Any-related errors.

I guess it can't hurt though. I don't think having codecov fail based on mypy status makes sense -- someone must have already explicitly accepted the Anys (if that's what this is about).

[jakkdl]

I'm probably still voting against, but certainly not vetoing it or anything if others think it's fine.

[jakkdl]

though having 7 (this+1 failing) codecov entries in the summary list looks way overkill/cluttery

[webknjaz]

though having 7 (this+1 failing) codecov entries in the summary list looks way overkill/cluttery

@jakkdl try enabling the newish feature preview for the PR checks. With that, it collapses most of the statuses, only surfacing the failing ones by default.

[webknjaz]

@A5rocks

"type preciseness" is just tracking Anys, correct? I'm pretty sure we handle those explicitly with type ignore comments, having enabled plenty of Any-related errors.

I think that MyPy is not configured to be fully strict, though. This is why the reported type coverage is at 91.79%.

I guess it can't hurt though. I don't think having codecov fail based on mypy status makes sense -- someone must have already explicitly accepted the Anys (if that's what this is about).

Not exactly, I think there are some Any-related checks that aren't enabled by default. That's what I've discovered while researching this. Pretty sure it's these: https://github.com/ansible/awx-plugins/blob/53ebc59/.mypy.ini#L49-L52.

So yeah, it may be useful to have this check voting.

@jakkdl and @A5rocks check out this view: https://app.codecov.io/gh/python-trio/trio/commit/90aa0739d926abd4d6ac0a897557f949a1aee1e6/tree/src/trio?flags%5B0%5D=MyPy. It reveals a ton of imprecise coverage.