Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prefer $EBPYTHONPREFIXES over $PYTHONPATH #4496

Closed
wants to merge 17 commits into from
Closed

Prefer $EBPYTHONPREFIXES over $PYTHONPATH #4496

Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
0ec2df5
Make new safe module abstraction for update_paths
Micket Apr 3, 2024
74a3cb3
Replace PYTHONPATHs with EBPYTHONPREFIX when possible
Micket Apr 3, 2024
03e304e
Add option prefer-ebpythonprefix-over-pythonpath
Micket Apr 4, 2024
b07ea3f
Add warning for duplicated PYTHONPATH's that should not occur
Micket Apr 9, 2024
8c66e1a
Add tests for PYTHONPATH to EBPYTHONPREFIX rewrite
Micket Apr 10, 2024
ccd3d7e
Move check for faulty PYTHONPATHs to always run it
Micket Apr 11, 2024
f76f320
Move conversion of paths as str to lists earlier removing duplicated …
Micket Apr 11, 2024
8b5389f
Fix name EBPYTHONPREFIX to EBPYTHONPREFIXES which is used in sitecust…
Micket Apr 11, 2024
cb6adb0
Move _filter_paths check to _update_paths
Micket Apr 12, 2024
4a11f2f
Simplify logic for _filter_paths
Micket Apr 14, 2024
c471e03
Rename flag to replace_pythonpath
Micket Apr 21, 2024
2676e4e
Add assertion for input argument to _filter_paths.
Micket Apr 21, 2024
f4fbe76
Improve typing hints for list of str
Micket Apr 21, 2024
92f0e62
Also use List from typing
Micket Apr 21, 2024
e91186e
Improve typing hints for list of str
Micket Apr 21, 2024
236c67f
Improve typing hints for list of str
Micket Apr 21, 2024
ca304fb
Update easybuild/tools/module_generator.py
Micket Apr 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 6 additions & 13 deletions easybuild/tools/module_generator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,25 +213,20 @@ def _filter_paths(self, key, paths):
return paths

added_paths = self.added_paths_per_key.setdefault(key, set())
# Coerce any iterable/generator into a list
if not isinstance(paths, list):
paths = list(paths)
Micket marked this conversation as resolved.
Show resolved Hide resolved
filtered_paths = [x for x in paths if x not in added_paths and not added_paths.add(x)]
if filtered_paths != paths:
removed_paths = paths if filtered_paths is None else [x for x in paths if x not in filtered_paths]
print_warning("Suppressed adding the following path(s) to $%s of the module as they were already added: %s",
Micket marked this conversation as resolved.
Show resolved Hide resolved
key, ', '.join(removed_paths),
log=self.log)
if not filtered_paths:
filtered_paths = None
return filtered_paths

def append_paths(self, key, paths, allow_abs=False, expand_relpaths=True):
"""
Generate append-path statements for the given list of paths.

:param key: environment variable to append paths to
:param paths: list of paths to append
:param paths: single or list of paths to append
:param allow_abs: allow providing of absolute paths
:param expand_relpaths: expand relative paths into absolute paths (by prefixing install dir)
"""
Expand All @@ -242,7 +237,7 @@ def prepend_paths(self, key, paths, allow_abs=False, expand_relpaths=True):
Generate prepend-path statements for the given list of paths.

:param key: environment variable to append paths to
:param paths: list of paths to append
:param paths: single or list of paths to append
:param allow_abs: allow providing of absolute paths
:param expand_relpaths: expand relative paths into absolute paths (by prefixing install dir)
"""
Expand All @@ -253,13 +248,15 @@ def update_paths(self, key, paths, prepend=True, allow_abs=False, expand_relpath
Generate append/prepend-path statements for the given list of paths.

:param key: environment variable to append paths to
:param paths: list of paths to append
:param paths: single or list of paths to append
:param allow_abs: allow providing of absolute paths
:param expand_relpaths: expand relative paths into absolute paths (by prefixing install dir)
"""
if isinstance(paths, str):
self.log.debug("Wrapping %s into a list before using it for %s", paths, key)
paths = [paths]
elif not isinstance(paths, list):
paths = list(paths) # coerce any iterator into a list

if key == 'PYTHONPATH':
python_paths = [path for path in paths if re.match(r'lib/python\d+\.\d+/site-packages', path)]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checking the logic in _filter_paths I see an issue here: You iterate paths multiple times: Here and at https://github.com/easybuilders/easybuild-framework/pull/4496/files#diff-4af72e9777d353325a29df400ab4229e14defd653e413db51786dc2172d3baefR275 but inside _update_paths you call _filter_paths which "coerces a generator into a list" if it isn't a list yet.

This means it this point paths might be a generator which can only be iterated once as otherwise the code in _filter_paths wasn't required.

Similar to my previous reasoning about breaking change in favor of cleaner code: I would simply disallow passing anything but a a string or a list to update_paths and assert that it is a list in _filter_paths instead of trying to convert a potentially exhausted generator to a list here or there.

Quick example:

paths=(i for i in range(10))
[i for i in paths if i==2] # [2]
[i for i in paths if i==2] # []

Micket marked this conversation as resolved.
Show resolved Hide resolved
Expand Down Expand Up @@ -976,14 +973,12 @@ def _update_paths(self, key, paths, prepend=True, allow_abs=False, expand_relpat
Generate prepend-path or append-path statements for the given list of paths.

:param key: environment variable to prepend/append paths to
:param paths: list of paths to prepend
:param paths: list of paths to prepend/append
Micket marked this conversation as resolved.
Show resolved Hide resolved
:param prepend: whether to prepend (True) or append (False) paths
:param allow_abs: allow providing of absolute paths
:param expand_relpaths: expand relative paths into absolute paths (by prefixing install dir)
"""
paths = self._filter_paths(key, paths)
if paths is None:
return ''

if prepend:
update_type = 'prepend'
Expand Down Expand Up @@ -1453,8 +1448,6 @@ def _update_paths(self, key, paths, prepend=True, allow_abs=False, expand_relpat
:param expand_relpaths: expand relative paths into absolute paths (by prefixing install dir)
"""
paths = self._filter_paths(key, paths)
if paths is None:
return ''

if prepend:
update_type = 'prepend'
Expand Down
Loading