Crash when there a "._xxx.pth" on macOS

[ronaldoussoren]

Bug report

Bug description:

See https://discuss.python.org/t/venv-not-function-properly-in-macos-within-an-exfat-disk/26145 for the original report.

MacOS creates "._" files with additional metadata when there is metadata (such as extended attributes) that the filesystem doesn't support natively.

To reproduce (I did this on macOS 13 to match the forum message, likely also happens on 14):

1. Create a disk image with the exFAT format and mount this

2. Create a virtual environment on this volume and active this

3. Install setup tools (which ships with a PTH file name distutils-precedence.pth)

4. Use the "Get Info" context menu in the finder for the PTH file and add a note to the metadata

   There should now be a ._distutils-precedence.pth file

   Alternatively: use xattr -w name value distutils-precedence.pth to write an extended attribute.

5. Start "python"
   This should fail with a UnicodeDecodeError in site.py

CPython versions tested on:

3.12

Operating systems tested on:

macOS

Linked PRs

• gh-113356: Ignore errors in "._ABC.pth" #113357

[quul]

Thank you for your attention! I am the one who report this.
I am now in macOS 14.2 (23C64), and I could reproduce the issue without manually tweaking(But obviously it doesn't effect the PR, just FYI).

Python version: Python 3.11.6 (main, Nov 2 2023, 04:39:43) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin

Reproduce steps:

1. Format an extra disk with exFAT
2. Create a venv with python3 -m venv venv
3. Enter the venv via source venv/bin/activate
4. Try to install anything with pip. For example pip install django, and then the error cames out

The ._ file appears after step 2:

╰─$ ls -al venv/lib/python3.11/site-packages
total 4096
drwx------@ 1 lincw  staff  131072 Dec 21 22:46 .
drwx------@ 1 lincw  staff  131072 Dec 21 22:46 ..
-rwx------  1 lincw  staff    4096 Dec 21 22:46 .__distutils_hack
-rwx------  1 lincw  staff    4096 Dec 21 22:46 ._distutils-precedence.pth
-rwx------  1 lincw  staff    4096 Dec 21 22:46 ._pip
-rwx------  1 lincw  staff    4096 Dec 21 22:46 ._pip-23.3.1.dist-info
-rwx------  1 lincw  staff    4096 Dec 21 22:46 ._pkg_resources
-rwx------  1 lincw  staff    4096 Dec 21 22:46 ._setuptools
-rwx------  1 lincw  staff    4096 Dec 21 22:46 ._setuptools-68.2.2.dist-info
drwx------@ 1 lincw  staff  131072 Dec 21 22:46 _distutils_hack
-rwx------@ 1 lincw  staff     151 Dec 21 22:46 distutils-precedence.pth
drwx------@ 1 lincw  staff  131072 Dec 21 22:46 pip
drwx------@ 1 lincw  staff  131072 Dec 21 22:46 pip-23.3.1.dist-info
drwx------@ 1 lincw  staff  131072 Dec 21 22:46 pkg_resources
drwx------@ 1 lincw  staff  131072 Dec 21 22:46 setuptools
drwx------@ 1 lincw  staff  131072 Dec 21 22:46 setuptools-68.2.2.dist-info

Sorry for anything bad since I'm not so good at English. :(

[ronaldoussoren]

Thank you for your attention! I am the one who report this. I am now in macOS 14.2 (23C64), and I could reproduce the issue without manually tweaking(But obviously it doesn't effect the PR, just FYI).

[...]

Sorry for anything bad since I'm not so good at English. :(

Thanks for the update. And don't worry about your English, it is perfectly clear.

[ronaldoussoren]

@quul, What's the output of the following commands (before removing the "._" files):

xattr -l venv/lib/python3.11/site-packages/distutils-precedence.pth
ls -le@ venv/lib/python3.11/site-packages/distutils-precedence.pth

Not really important, but I'm curious what metadata is added on your system. I don't have an external disk lying around that I can format exFAT, otherwise I'd test myself.

[serhiy-storchaka]

@ronaldoussoren, you perhaps can create a file system in a file and them mount it. At least it is easy to do on Linux:

$ dd of=exfat.img if=/dev/null bs=1G count=0 seek=1
$ mkfs.exfat exfat.img
$ mkdir mountdir
$ sudo mount exfat.img mountdir

We can ignore UnicodeDecodeError as well as OSError when open a pth file (although the code will be more complex to avoid leaks), but I think that it is reasonable to skip dot-files when list pth files at first place.

[ronaldoussoren]

@ronaldoussoren, you perhaps can create a file system in a file and them mount it. At least it is easy to do on Linux:

$ dd of=exfat.img if=/dev/null bs=1G count=0 seek=1
$ mkfs.exfat exfat.img
$ mkdir mountdir
$ sudo mount exfat.img mountdir

We can ignore UnicodeDecodeError as well as OSError when open a pth file (although the code will be more complex to avoid leaks), but I think that it is reasonable to skip dot-files when list pth files at first place.

I can reproduce the issue using a disk image, but only by manually settings some extended attributes. Users on the d.p.o thread I link to run into this without doing anything special. The only difference is that they use an external disk.

[quul]

@quul, What's the output of the following commands (before removing the "._" files):

xattr -l venv/lib/python3.11/site-packages/distutils-precedence.pth
ls -le@ venv/lib/python3.11/site-packages/distutils-precedence.pth

Not really important, but I'm curious what metadata is added on your system. I don't have an external disk lying around that I can format exFAT, otherwise I'd test myself.

I’m glad to do the test. However I just started my vocation and not carrying the mac with me.
I’ll do this as soon as I come back to office(perhaps 1 week+)

[ronaldoussoren]

This has been fixed by #113659, which will ignore all hidden PTH files, including those with a filename starting with a dot like the ._ files seen in this issue.