Can we use mypy_primer to check the effects of adding a py.typed marker or library changes?

[sirosen]

Use-case / Context

Suppose I have a library which is already in typeshed and which has started to add annotations.
Comparing my annotations with typeshed is difficult until I publish a py.typed file. At that point, users may start to see new or different errors if the new annotations are incomplete or inaccurate.
Introducing improper type errors because the library annotations are wrong is undesirable.

mypy_primer is well-situated to help me because it compares and diffs results of two different "typing states". This is just another flavor of diff I want to produce.

A very closely related case is testing a change in a library using mypy_primer, when py.typed is already in use. Again, the diff from mypy_primer could be quite useful.

Question

How can I test the impact of adding a py.typed file using typeshed? Or how do I test the impact other library typing changes?

I've run mypy_primer locally a while back, I'm prepared to contribute support (I think some would be needed), &c.
I think the ideal workflow here is that I clone and modify my library to add the py.typed. Optionally, I build it to a wheel or sdist to prepare it as a package. And then I run some magic incantation:

mypy_primer -k myproject --add-install ~/myproject/myproject_0.1.whl

?

I'm trying to think through ways of doing this. I could run a local package index and then flag it with pip options, I think?
But that seems very fiddly for everyone to work out themselves.
If that path would work and seems desirable, I'd love to use this issue to hammer out how to do it and then PR with a doc on the process.

Please let me know if you think this problem is in or out of scope for mypy_primer, and if it is in, what you think is the best solution.

[hauntsaninja]

I'd added a --new-mypypath and --old-mypypath for this kind of use case. It lets you put different things on MYPYPATH for the "new" run and the "old" run. MYPYPATH can be a little funky, but seems like you could get it to work for this use case

[sirosen]

I realized it's been a couple of weeks... Sorry about being slow on my end.

Thanks for the pointer/clarification. That seems like it will work but will require some setup. I'll try to find time to play with it.

It's it worth documenting once it's figured out as a complete workflow? I think it's a cool and interesting use case but I have no clear idea where it should be written down.

[A5rocks]

Hi, I just realized this would be really useful for a library I help maintain! I think this would be worth documenting.