Make _EditableFinder case-sensitive on case-insensitive filesystems (for certain editable installations)

[aganders3]

Summary of changes

This updates the _EditableFinder that is added to sys.meta_path for certain editable installations to respect filename case on case-insensitive (but case-preserving) filesystems. I believe this behavior better matches normal installations and PEP 235.

The way the change works is by checking the path is in the directory listing (which returns proper case) in addition to checking if the file exists (which is generally case-insensitive if the filesystem is). I believe this is pretty much how it is done in the FileFinder in the standard library (see here and here for example).

Closes #3994

Pull Request Checklist

• Changes have tests
• News fragment added in newsfragments/.
  (See documentation for details)

[aganders3]

It's not terribly surprising to see this Windows failure. Unfortunately I don't have very easy access to a Windows system but I'll see what I can do.

[aganders3]

I pushed 07d0242 to hopefully address the lint errors in the docs build -- I'm not sure why these errors showed up here. Maybe there was a new release of sphinx-lint or something. Let me know if you'd rather see that commit moved to a different PR.

[abravalheri]

Hi @aganders3, thank you very much for looking into this (including the changes in the docs).
Please see some comments below.

[aganders3]

Unfortunately I still couldn't make this work with public classes from importlib.machinery. It mostly worked for normal imports, but needed enough special logic for namespace packages that just expanding the previous method ended up being simpler. I can keep trying along those lines if you'd prefer, but I don't immediately see a clean solution.

I expanded the tests here to cover more of the cases you brought up. Thanks for the review!

[abravalheri]

Hi @aganders3 thank you very much for working on this.
Sorry I haven't had the time to review this PR in detail.
My initial/preliminary thoughts are:

• I plan to split the commits targeting the docs linter (thank you very much for that!) into a separated PR. This way I can merge the docs part and rebase this PR, which will make it a little cleaner and easier to review.
• I am concerned with the increase in complexity for the finder. I am also thinking about the loops to check file/directory existence in the file system, since they might make import statements slow (with only 1 project installed as editable, that should be fine, but maybe if the user installs multiple editable projects in the same virtualenv it might became a problem).
  • I was planning to experiment a bit with the idea I expressed in Make _EditableFinder case-sensitive on case-insensitive filesystems (for certain editable installations) #3995 (comment), and/or delegating calls to importlib.machinery.FileFinder or importlib.machinery.PathFinder (they already implement the necessary case sensitive checks and caching to not hurt performance, so it would be very nice if we can simply use them)

Once I have a bit more time I can work on this, but please feel free to bet me to that.
Regardless, thank you very much for the contribution (that includes tests ❤️), even if we don't add exactly the code as it is right now, I am sure we are going to reuse most of it in the implementation.

[aganders3]

No worries - thanks again for the reviews! I may spend a little more time looking again at using importlib.machinery but otherwise will probably leave this alone for the moment. At the very least I am happy to see this issue being addressed and I will walk away with a much deeper understanding of the import system 😅.

[abravalheri]

@aganders3, I submitted a few commits trying something out...
If it does not work we can revert the commits.

[aganders3]

Nice! It at least passes my tests locally (on macOS). _find_nested_spec is what I couldn't quite work out, but this looks very clean. I had not considered using spec_from_file_location along with a spec already discovered from a finder.

Thanks!

edit: I also tried your solution with my test project and it seems to be working as expected

[abravalheri]

Thank you very much