[BUG] Editable Installation with src-layout Fails Due to Hidden '.venv'

[fkhn]

setuptools version

setuptools==65.5.0

Python version

Python 3.11

OS

macOS

Additional environment information

a venv name '.venv'
pip 24.2

Description

When using pip install -e . with a src-layout, an entry is typically added to sys.path via a file named __editable__.pakagename.pth. However, the loading of this .pth file fails if the virtual environment is hidden, which is the case for the default .venv name used by many tools.
To be precise, cpython/Lib/site.py uses os.lstat on the .pth file and skips it.
It is likely the same on other Unix-like systems.

Expected behavior

Since src-layout and editable installs are frequently used, they should be supported regardless of the virtual environment's naming convention or visibility status.

How to Reproduce

create a venv with a name starting by a dot, typically .venv
use a src-layout
python3 -v -m packagename

Output

Skipping hidden .pth file: '/project_folder/.venv/lib/python3.11/site-packages/__editable__.packagename.pth'

[abravalheri]

Please note that users are advised to try the last version available of setuptools before reporting problems. Setuptools development model only releases monotonically increasing versions, so if any change is necessary to solve the problem it is only done on top of the main branch.

---

I am not sure why you are getting this message. The file that setuptools produces is not a hidden file itself: it does not start with a . and we do not add any special attribute to it when creating the wheel used by pip, so there is nothing we can really change about it.

The problem description also mentions "likely the same on other Unix-like systems", but I cannot reproduce this in a different POSIX environment:

# docker run --rm -it python:3.11-bookworm /bin/bash
mkdir -p /tmp/proj/src
cd /tmp/proj

cat <<EOF > pyproject.toml
[build-system]
requires = ["setuptools==73.0.1"]
build-backend = "setuptools.build_meta"
EOF

cat <<EOF > src/pkg.py
print("hello world")
EOF

python -m venv .venv
.venv/bin/python -m pip install -U "pip==24.2"
.venv/bin/python -m pip install -e .
# ...
# Successfully built pkg
# Installing collected packages: pkg
# Successfully installed pkg-0.0.0

ls -la .venv/lib/python3.11/site-packages/__editable__.pkg-0.0.0.pth
# -rw-r--r-- 1 root root 14 Aug 26 10:35 .venv/lib/python3.11/site-packages/__editable__.pkg-0.0.0.pth

.venv/bin/python -m pkg
# hello world

.venv/bin/python -v -m pkg

import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'posix' # <class '_frozen_importlib.BuiltinImporter'>
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
# installing zipimport hook
import 'time' # <class '_frozen_importlib.BuiltinImporter'>
import 'zipimport' # <class '_frozen_importlib.FrozenImporter'>
# installed zipimport hook
# /usr/local/lib/python3.11/encodings/__pycache__/__init__.cpython-311.pyc matches /usr/local/lib/python3.11/encodings/__init__.py
# code object from '/usr/local/lib/python3.11/encodings/__pycache__/__init__.cpython-311.pyc'
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import 'codecs' # <class '_frozen_importlib.FrozenImporter'>
# /usr/local/lib/python3.11/encodings/__pycache__/aliases.cpython-311.pyc matches /usr/local/lib/python3.11/encodings/aliases.py
# code object from '/usr/local/lib/python3.11/encodings/__pycache__/aliases.cpython-311.pyc'
import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x7f6a92609550>
import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x7f6a925f5990>
# /usr/local/lib/python3.11/encodings/__pycache__/utf_8.cpython-311.pyc matches /usr/local/lib/python3.11/encodings/utf_8.py
# code object from '/usr/local/lib/python3.11/encodings/__pycache__/utf_8.cpython-311.pyc'
import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x7f6a925f7350>
import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
import '_abc' # <class '_frozen_importlib.BuiltinImporter'>
import 'abc' # <class '_frozen_importlib.FrozenImporter'>
import 'io' # <class '_frozen_importlib.FrozenImporter'>
import '_stat' # <class '_frozen_importlib.BuiltinImporter'>
import 'stat' # <class '_frozen_importlib.FrozenImporter'>
import '_collections_abc' # <class '_frozen_importlib.FrozenImporter'>
import 'genericpath' # <class '_frozen_importlib.FrozenImporter'>
import 'posixpath' # <class '_frozen_importlib.FrozenImporter'>
import 'os' # <class '_frozen_importlib.FrozenImporter'>
import '_sitebuiltins' # <class '_frozen_importlib.FrozenImporter'>
Processing global site-packages
Adding directory: '/tmp/proj/.venv/lib/python3.11/site-packages'
Processing .pth file: '/tmp/proj/.venv/lib/python3.11/site-packages/__editable__.pkg-0.0.0.pth'
Processing .pth file: '/tmp/proj/.venv/lib/python3.11/site-packages/distutils-precedence.pth'
# /tmp/proj/.venv/lib/python3.11/site-packages/_distutils_hack/__pycache__/__init__.cpython-311.pyc matches /tmp/proj/.venv/lib/python3.11/site-packages/_distutils_hack/__init__.py
# code object from '/tmp/proj/.venv/lib/python3.11/site-packages/_distutils_hack/__pycache__/__init__.cpython-311.pyc'
import '_distutils_hack' # <_frozen_importlib_external.SourceFileLoader object at 0x7f6a923d8350>
Processing user site-packages
Processing global site-packages
Adding directory: '/tmp/proj/.venv/lib/python3.11/site-packages'
Processing .pth file: '/tmp/proj/.venv/lib/python3.11/site-packages/__editable__.pkg-0.0.0.pth'
Processing .pth file: '/tmp/proj/.venv/lib/python3.11/site-packages/distutils-precedence.pth'
import 'site' # <class '_frozen_importlib.FrozenImporter'>
Python 3.11.9 (main, Apr  3 2024, 23:59:48) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
# /usr/local/lib/python3.11/importlib/__pycache__/__init__.cpython-311.pyc matches /usr/local/lib/python3.11/importlib/__init__.py
# code object from '/usr/local/lib/python3.11/importlib/__pycache__/__init__.cpython-311.pyc'
# /usr/local/lib/python3.11/__pycache__/warnings.cpython-311.pyc matches /usr/local/lib/python3.11/warnings.py
# code object from '/usr/local/lib/python3.11/__pycache__/warnings.cpython-311.pyc'
import 'warnings' # <_frozen_importlib_external.SourceFileLoader object at 0x7f6a923da2d0>
import 'importlib' # <_frozen_importlib_external.SourceFileLoader object at 0x7f6a923d9ed0>
import 'importlib.machinery' # <class '_frozen_importlib.FrozenImporter'>
# /usr/local/lib/python3.11/importlib/__pycache__/_abc.cpython-311.pyc matches /usr/local/lib/python3.11/importlib/_abc.py
# code object from '/usr/local/lib/python3.11/importlib/__pycache__/_abc.cpython-311.pyc'
import 'importlib._abc' # <_frozen_importlib_external.SourceFileLoader object at 0x7f6a923f0710>
# /usr/local/lib/python3.11/__pycache__/contextlib.cpython-311.pyc matches /usr/local/lib/python3.11/contextlib.py
# code object from '/usr/local/lib/python3.11/__pycache__/contextlib.cpython-311.pyc'
# /usr/local/lib/python3.11/collections/__pycache__/__init__.cpython-311.pyc matches /usr/local/lib/python3.11/collections/__init__.py
# code object from '/usr/local/lib/python3.11/collections/__pycache__/__init__.cpython-311.pyc'
import 'itertools' # <class '_frozen_importlib.BuiltinImporter'>
# /usr/local/lib/python3.11/__pycache__/keyword.cpython-311.pyc matches /usr/local/lib/python3.11/keyword.py
# code object from '/usr/local/lib/python3.11/__pycache__/keyword.cpython-311.pyc'
import 'keyword' # <_frozen_importlib_external.SourceFileLoader object at 0x7f6a9241e1d0>
# /usr/local/lib/python3.11/__pycache__/operator.cpython-311.pyc matches /usr/local/lib/python3.11/operator.py
# code object from '/usr/local/lib/python3.11/__pycache__/operator.cpython-311.pyc'
import '_operator' # <class '_frozen_importlib.BuiltinImporter'>
import 'operator' # <_frozen_importlib_external.SourceFileLoader object at 0x7f6a9241ec90>
# /usr/local/lib/python3.11/__pycache__/reprlib.cpython-311.pyc matches /usr/local/lib/python3.11/reprlib.py
# code object from '/usr/local/lib/python3.11/__pycache__/reprlib.cpython-311.pyc'
import 'reprlib' # <_frozen_importlib_external.SourceFileLoader object at 0x7f6a9241f950>
import '_collections' # <class '_frozen_importlib.BuiltinImporter'>
import 'collections' # <_frozen_importlib_external.SourceFileLoader object at 0x7f6a923f1910>
# /usr/local/lib/python3.11/__pycache__/functools.cpython-311.pyc matches /usr/local/lib/python3.11/functools.py
# code object from '/usr/local/lib/python3.11/__pycache__/functools.cpython-311.pyc'
# /usr/local/lib/python3.11/__pycache__/types.cpython-311.pyc matches /usr/local/lib/python3.11/types.py
# code object from '/usr/local/lib/python3.11/__pycache__/types.cpython-311.pyc'
import 'types' # <_frozen_importlib_external.SourceFileLoader object at 0x7f6a92442a10>
import '_functools' # <class '_frozen_importlib.BuiltinImporter'>
import 'functools' # <_frozen_importlib_external.SourceFileLoader object at 0x7f6a92400990>
import 'contextlib' # <_frozen_importlib_external.SourceFileLoader object at 0x7f6a923da190>
# /usr/local/lib/python3.11/__pycache__/threading.cpython-311.pyc matches /usr/local/lib/python3.11/threading.py
# code object from '/usr/local/lib/python3.11/__pycache__/threading.cpython-311.pyc'
# /usr/local/lib/python3.11/__pycache__/_weakrefset.cpython-311.pyc matches /usr/local/lib/python3.11/_weakrefset.py
# code object from '/usr/local/lib/python3.11/__pycache__/_weakrefset.cpython-311.pyc'
import '_weakrefset' # <_frozen_importlib_external.SourceFileLoader object at 0x7f6a92461610>
import 'threading' # <_frozen_importlib_external.SourceFileLoader object at 0x7f6a923f18d0>
import 'importlib.util' # <class '_frozen_importlib.FrozenImporter'>
import 'runpy' # <class '_frozen_importlib.FrozenImporter'>
# /tmp/proj/src/__pycache__/pkg.cpython-311.pyc matches /tmp/proj/src/pkg.py
# code object from '/tmp/proj/src/__pycache__/pkg.cpython-311.pyc'
hello world
# clear builtins._
# clear sys.path
# clear sys.argv
# clear sys.ps1
# clear sys.ps2
# clear sys.last_type
# clear sys.last_value
# clear sys.last_traceback
# clear sys.path_hooks
# clear sys.path_importer_cache
# clear sys.meta_path
# clear sys.__interactivehook__
# restore sys.stdin
# restore sys.stdout
# restore sys.stderr
# cleanup[2] removing sys
# cleanup[2] removing builtins
# cleanup[2] removing _frozen_importlib
# cleanup[2] removing _imp
# cleanup[2] removing _thread
# cleanup[2] removing _warnings
# cleanup[2] removing _weakref
# cleanup[2] removing _io
# cleanup[2] removing marshal
# cleanup[2] removing posix
# cleanup[2] removing _frozen_importlib_external
# cleanup[2] removing time
# cleanup[2] removing zipimport
# destroy zipimport
# cleanup[2] removing _codecs
# cleanup[2] removing codecs
# cleanup[2] removing encodings.aliases
# cleanup[2] removing encodings
# destroy encodings
# cleanup[2] removing encodings.utf_8
# cleanup[2] removing _signal
# cleanup[2] removing _abc
# cleanup[2] removing abc
# cleanup[2] removing io
# cleanup[2] removing __main__
# destroy __main__
# cleanup[2] removing _stat
# cleanup[2] removing stat
# cleanup[2] removing _collections_abc
# cleanup[2] removing genericpath
# cleanup[2] removing posixpath
# cleanup[2] removing os.path
# cleanup[2] removing os
# cleanup[2] removing _sitebuiltins
# cleanup[2] removing _distutils_hack
# destroy _distutils_hack
# cleanup[2] removing site
# destroy site
# cleanup[2] removing importlib._bootstrap
# cleanup[2] removing importlib._bootstrap_external
# cleanup[2] removing warnings
# cleanup[2] removing importlib
# cleanup[2] removing importlib.machinery
# cleanup[2] removing importlib._abc
# cleanup[2] removing itertools
# cleanup[2] removing keyword
# destroy keyword
# cleanup[2] removing _operator
# cleanup[2] removing operator
# destroy operator
# cleanup[2] removing reprlib
# destroy reprlib
# cleanup[2] removing _collections
# cleanup[2] removing collections
# destroy collections
# cleanup[2] removing types
# cleanup[2] removing _functools
# cleanup[2] removing functools
# cleanup[2] removing contextlib
# destroy contextlib
# cleanup[2] removing _weakrefset
# destroy _weakrefset
# cleanup[2] removing threading
# cleanup[2] removing importlib.util
# cleanup[2] removing runpy
# destroy runpy
# destroy _signal
# destroy _sitebuiltins
# destroy importlib
# destroy io
# destroy importlib.machinery
# destroy importlib._abc
# destroy importlib.util
# destroy threading
# destroy types
# destroy warnings
# destroy itertools
# cleanup[3] wiping functools
# cleanup[3] wiping _functools
# cleanup[3] wiping _collections
# cleanup[3] wiping _operator
# cleanup[3] wiping importlib._bootstrap_external
# cleanup[3] wiping importlib._bootstrap
# cleanup[3] wiping os
# destroy abc
# destroy posixpath
# cleanup[3] wiping genericpath
# cleanup[3] wiping _collections_abc
# cleanup[3] wiping stat
# cleanup[3] wiping _stat
# destroy _stat
# cleanup[3] wiping _abc
# cleanup[3] wiping encodings.utf_8
# cleanup[3] wiping encodings.aliases
# cleanup[3] wiping codecs
# cleanup[3] wiping _codecs
# cleanup[3] wiping time
# cleanup[3] wiping _frozen_importlib_external
# cleanup[3] wiping posix
# cleanup[3] wiping marshal
# cleanup[3] wiping _io
# cleanup[3] wiping _weakref
# cleanup[3] wiping _warnings
# cleanup[3] wiping _thread
# cleanup[3] wiping _imp
# cleanup[3] wiping _frozen_importlib
# cleanup[3] wiping sys
# cleanup[3] wiping builtins
# destroy _frozen_importlib_external
# destroy _imp
# destroy io
# destroy marshal
# destroy _warnings
# destroy stat
# destroy genericpath
# destroy posix
# destroy _collections_abc
# destroy _weakref
# destroy _collections
# destroy _abc
# destroy _functools
# destroy _operator
# destroy _frozen_importlib
# destroy codecs
# destroy encodings.aliases
# destroy encodings.utf_8
# destroy _codecs
# destroy os
# destroy functools
# destroy time
# destroy sys
# destroy builtins
# destroy _thread
# clear sys.audit hooks

In Linux I cannot see any Skipping hidden .pth file message. On the contrary, we see Processing .pth file: '/tmp/proj/.venv/lib/python3.11/site-packages/__editable__.pkg-0.0.0.pth'.

Based on this code (from python/cpython@21ab20b), we can also check inside of the container for any attributes: https://github.com/python/cpython/blob/v3.11.9/Lib/site.py#L172-L180

# .venv/bin/python
Python 3.11.9 (main, Apr  3 2024, 23:59:48) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, stat
>>> st = os.lstat('.venv/lib/python3.11/site-packages/__editable__.pkg-0.0.0.pth')
>>> st
os.stat_result(st_mode=33188, st_ino=200282, st_dev=230, st_nlink=1, st_uid=0, st_gid=0, st_size=14, st_atime=1724668626, st_mtime=1724668530, st_ctime=1724668530)
>>> getattr(st, 'st_flags', 0) & stat.UF_HIDDEN
0
>>> getattr(st, 'st_file_attributes', 0) & stat.FILE_ATTRIBUTE_HIDDEN
0

That said, this is the furthest I can go with this investigation, because I don't have access to a macOS machine. If anyone would like to take the investigation further and find out the origin of the problem that would be very nice.

For now the problem sounds very weird:

There is nothing in the setuptools code adding any kind of flag or file attribute to make the file hidden.

Also in the code https://github.com/python/cpython/blob/v3.11.9/Lib/site.py#L229-L234 we can see that site.py uses os.listdir, which I believe should return basenames without the parent directory prefix, so the starting . would not be present there...

[fkhn]

Your comment leads me to a specific macOS stuff. My folder is under Documents and this one is synchronized with iCloud. The side effect of that is that the hidden flag is set by that machinery. So now we know it’s not as wide as Unix-like but narrowed to macOS with iCloud sync. A better situation as fewer people are concerned, but still annoying.

[fkhn]

To reproduce the macOS iCloud behavior, I have setup a test folder with both dot and not dot folder and move that test folder under Documents that is synched: see the result on hidden flag.

find test_hidden -exec ls -ldO \{\} \;
drwxr-xr-x  5 franck  staff  - 160 26 aoû 16:30 test_hidden
drwxr-xr-x  3 franck  staff  - 96 26 aoû 16:16 test_hidden/.dot_folder
drwxr-xr-x  3 franck  staff  - 96 26 aoû 16:16 test_hidden/.dot_folder/inner_folder
-rw-r--r--  1 franck  staff  - 0 26 aoû 14:36 test_hidden/.dot_folder/inner_folder/a_file
-rw-r--r--  1 franck  staff  - 618 26 aoû 14:52 test_hidden/pyproject.toml
drwxr-xr-x  3 franck  staff  - 96 26 aoû 16:16 test_hidden/nodot_folder
drwxr-xr-x  3 franck  staff  - 96 26 aoû 16:16 test_hidden/nodot_folder/inner_folder
-rw-r--r--  1 franck  staff  - 0 26 aoû 14:37 test_hidden/nodot_folder/inner_folder/a_file

mv test_hidden Documents

find Documents/test_hidden -exec ls -ldO \{\} \; 
drwxr-xr-x  5 franck  staff  - 160 26 aoû 16:30 Documents/test_hidden
drwxr-xr-x  3 franck  staff  hidden 96 26 aoû 16:30 Documents/test_hidden/.dot_folder
drwxr-xr-x  3 franck  staff  hidden 96 26 aoû 16:30 Documents/test_hidden/.dot_folder/inner_folder
-rw-r--r--  1 franck  staff  hidden 0 26 aoû 14:36 Documents/test_hidden/.dot_folder/inner_folder/a_file
-rw-r--r--  1 franck  staff  - 618 26 aoû 14:52 Documents/test_hidden/pyproject.toml
drwxr-xr-x  3 franck  staff  - 96 26 aoû 16:30 Documents/test_hidden/nodot_folder
drwxr-xr-x  3 franck  staff  - 96 26 aoû 16:30 Documents/test_hidden/nodot_folder/inner_folder
-rw-r--r--  1 franck  staff  - 0 26 aoû 14:37 Documents/test_hidden/nodot_folder/inner_folder/a_file

With that in mind, it is a macOS or python subject more than a setuptools one. I guess that there is other contexts where the hidden flag might be set and trigger the same trouble. If setuptools might help either on the hidden python policy for pth file, or to warn users. That was quite difficult to isolate.

[abravalheri]

If setuptools might help either on the hidden python policy for pth file, or to warn users. That was quite difficult to isolate.

I am afraid setuptools cannot help with that, because the development of Python happens very separated from setuptools and vice versa... Python policy for .pth files is something that has to be discussed with the Python core devs either in https://discuss.python.org or via an issue/discussion in the https://github.com/python/cpython repository.

We are not in a very good position to warn the users either, because setuptools only builds a wheel file and does not know where it is going to be installed... That information is under the control of installers like pip. Moreover, installers usually hide all output from setuptools, even warnings. Users don't see any of the messages we have in editable installations. For example I don't think any user ever saw the following warnings.warn message for non-src-layout editable installations: https://github.com/pypa/setuptools/blob/v73.0.1/setuptools/command/editable_wheel.py#L531-L535; or the following logging.warning message for src-layouts: https://github.com/pypa/setuptools/blob/v73.0.1/setuptools/command/editable_wheel.py#L399-L403.

[abravalheri]

Thank you very much @fkhn for the investigation and finding out what is the root of the problem.
Since we identified that there is not much setuptools can do for this case (everything seems to be outside of the control of setuptools) probably the best next step would be asking for the advice of the Python core devs about this situation.

The checks for hidden files seem to be a new "feature" in Python (merged at the beginning of this year): python/cpython#113660, so I am not sure if the core developers where already notified of this edge case where it is not the user (legitimate or malicious) that marks the .pth file as hidden, but iCloud itself (and not because of the name of the .pth file, but because of the name of a parent folder -- which is even worse). It may be worth discussing with them.

It is also important to point out 2 things:

• The behaviour that you observed does not seem to be restricted to editable installs or more specifically a setuptools editable installs. ANY .pth file on a .venv environment would be affected and skipped.
• .pth files are currently at the core of any implementation of editable installs. In the current state of Python's packaging ecosystem, it cannot be done consistently without them. So if there is some external factor making Python skip .venv/**/*.pth files, the ability of having editable installs in a specific virtual environment is fully compromised.

I will go ahead and close this issue because I don't think we can do anything in setuptools code base to get around this problem.

[fkhn]

Thank you very much @abravalheri for your reactivity and constructive comments that help to discover the root cause. I have appreciated your pointers to cpython. I am fine with the closure of the current issue. For your information I have added a comment on cpython/issues/113659 to make the link to the current discussion.