Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

update easyblock to let EasyBuild framework take care of prepend to $PYTHONPATH or $EBPYTHONPREFIXES + make PythonBundle and PythonPackage aware of --prefer-python-search-path EasyBuild configuration option #3343

Merged
merged 10 commits into from
Oct 2, 2024
Merged

update easyblock to let EasyBuild framework take care of prepend to $PYTHONPATH or $EBPYTHONPREFIXES + make PythonBundle and PythonPackage aware of --prefer-python-search-path EasyBuild configuration option #3343

merged 10 commits into from
Oct 2, 2024

Conversation

Micket
Copy link
Contributor

@Micket Micket commented May 29, 2024 •
edited by boegel
Loading

@Micket Micket added feature EasyBuild-5.0 EasyBuild 5.0 labels May 29, 2024
@Micket Micket added this to the 5.0 milestone May 29, 2024
@Micket Micket changed the title Use new prefer_ebpythonprefixes Use prefer_ebpythonprefixes flag to use EBPYTHONPREFIXES May 29, 2024
@Micket Micket mentioned this pull request Jul 7, 2024
Merged
@Micket
Copy link
Contributor Author

Micket commented Aug 5, 2024 •
edited
Loading

special easyblocks that have been directly modified:

  • amber - Amber-22.0-foss-2021b-AmberTools-22.3.eb - no source
  • cplex - CPLEX-22.1.1-GCCcore-11.2.0.eb - no source
  • gurobi - Gurobi-11.0.0-GCCcore-12.3.0.eb
  • lammps - LAMMPS-2Aug2023_update2-foss-2023a-kokkos.eb
  • mxnet - MXNet-1.9.1-foss-2022a.eb
  • openbabel - OpenBabel-3.1.1-gompi-2023a.eb
  • opencv - OpenCV-4.6.0-foss-2022a-contrib.eb
  • qscintilla - QScintilla-2.11.6-GCCcore-11.2.0.eb
  • tkinter - Tkinter-3.11.5-GCCcore-13.2.0.eb

PythonPackage or PythonBundle or subclasses indirectly modified:

  • Python - Python-3.11.5-GCCcore-13.2.0.eb
  • PythonPackage - PyYAML-6.0.1-GCCcore-13.2.0.eb
  • PythonBundle - Python-bundle-PyPI-2024.06-GCCcore-13.3.0.eb
  • CargoPythonPackage - cryptography-41.0.5-GCCcore-13.2.0.eb
  • CargoPythonBundle - pydantic-2.5.3-GCCcore-12.3.0.eb
  • wxpython - wxPython-4.2.1-foss-2022a.eb
  • dm_reverb - dm-reverb-0.7.0-foss-2021b.eb
  • torchvision - torchvision-0.16.0-foss-2023a.eb
  • tensorrt - TensorRT-8.6.1-foss-2022a-CUDA-11.7.0.eb
  • tensorflow - TensorFlow-2.9.1-foss-2022a.eb
  • tensorflow_compression - tensorflow-compression-2.11.0-foss-2022a-CUDA-11.7.0.eb
  • numpy - part of testing Scipy bundle
  • numexpr - numexpr-2.8.4-foss-2022a.eb
  • netcdf4_python - netcdf4-python-1.6.5-foss-2023b.eb
  • configuremakepythonpackage - PyQt5-5.15.7-GCCcore-12.2.0.eb
  • cmakepythonpackage - vigra-1.11.2-foss-2023a.eb
  • fortranpythonpackage - inherited by scipy easyblock
  • libxml2 - libxml2-2.12.7-GCCcore-13.3.0.eb
  • pybind11 - pybind11-2.11.1-GCCcore-13.2.0.eb
  • pyzmq - PyZMQ-25.1.2-GCCcore-13.2.0.eb (is this even using pyzmq? i don't think so)
  • sepp - SEPP-4.5.1-foss-2022b.eb
  • sympy - sympy-1.12-gfbf-2023b.eb
  • scipy - SciPy-bundle-2023.11-gfbf-2023b.eb
  • eqqlib - EggLib-3.3.0-GCC-13.2.0.eb
  • easybuildmeta - EasyBuild-4.9.2.eb
  • jaxlib - jax-0.4.4-foss-2022a.eb

Nothing to test:

  • vsc_tools - nothing?
  • cryptography - not in use anymore?
  • ufc - nothing
  • nose - nothing? nose3 uses pythonpackage directly.
  • dolfin - DOLFIN-xxx.eb (nothing remains)
  • versionindependentpythonpackage - vsc-base (nothing remains)
  • pyquante - PyQuante (nothing remains)
  • python_meep - python-meep (nothing remains)

(todo: make list of specific easyconfigs to test for the generic easyblocks)

@Micket
Copy link
Contributor Author

Micket commented Aug 15, 2024

Slight issue with libxml2; the easyblock inherits from PythonPackage, and PythonPackage has always unconditionally added PYTHONPATH/EBPYTHONPREFIXES, so that behavior is retained here.

I could change this; always let the framework add the environment variable if it detects the corresponding path, but this is not done here (yet?)

@Micket Micket force-pushed the ebpythonprefixes branch 4 times, most recently from 2ed73d5 to 1a4bd52 Compare August 15, 2024 22:23
@Micket
Copy link
Contributor Author

Micket commented Aug 22, 2024

Test report by @Micket

Overview of tested easyconfigs (in order)

  • SUCCESS Gurobi-11.0.0-GCCcore-12.3.0.eb
  • SUCCESS LAMMPS-2Aug2023_update2-foss-2023a-kokkos.eb
  • SUCCESS MXNet-1.9.1-foss-2022a.eb
  • SUCCESS OpenBabel-3.1.1-gompi-2023a.eb
  • SUCCESS OpenCV-4.6.0-foss-2022a-contrib.eb
  • SUCCESS QScintilla-2.11.6-GCCcore-11.2.0.eb
  • SUCCESS Tkinter-3.11.5-GCCcore-13.2.0.eb

Build succeeded for 7 out of 7 (7 easyconfigs in total)
alvis-skylake-build - Linux Rocky Linux 8.9, x86_64, Intel Xeon Processor (Skylake, IBRS, no TSX), Python 3.6.8
See https://gist.github.com/Micket/9eef93fc27bd9113f744e9a00c06ba64 for a full test report.

@Micket
Copy link
Contributor Author

Micket commented Aug 22, 2024

Test report by @Micket

Overview of tested easyconfigs (in order)

  • SUCCESS Python-3.11.5-GCCcore-13.2.0.eb
  • SUCCESS PyYAML-6.0.1-GCCcore-13.2.0.eb
  • SUCCESS Python-bundle-PyPI-2024.06-GCCcore-13.3.0.eb
  • SUCCESS cryptography-41.0.5-GCCcore-13.2.0.eb
  • SUCCESS pydantic-2.5.3-GCCcore-12.3.0.eb

Build succeeded for 5 out of 5 (5 easyconfigs in total)
alvis-skylake-build - Linux Rocky Linux 8.9, x86_64, Intel Xeon Processor (Skylake, IBRS, no TSX), Python 3.6.8
See https://gist.github.com/Micket/6f25e6b6852273bd08bb082839e84420 for a full test report.

@Micket
Copy link
Contributor Author

Micket commented Aug 22, 2024

Test report by @Micket

Overview of tested easyconfigs (in order)

  • SUCCESS wxPython-4.2.1-foss-2022a.eb
  • SUCCESS dm-reverb-0.7.0-foss-2021b.eb
  • SUCCESS torchvision-0.16.0-foss-2023a.eb
  • SUCCESS numexpr-2.8.4-foss-2022a.eb
  • SUCCESS PyQt5-5.15.7-GCCcore-12.2.0.eb
  • SUCCESS libxml2-2.12.7-GCCcore-13.3.0.eb
  • SUCCESS pybind11-2.11.1-GCCcore-13.2.0.eb

Build succeeded for 7 out of 7 (7 easyconfigs in total)
vera-skylake-build - Linux Rocky Linux 8.9, x86_64, Intel Xeon Processor (Skylake, IBRS, no TSX), Python 3.6.8
See https://gist.github.com/Micket/e60e25dd2a7ee72182ebded2138fdf1c for a full test report.

@Micket
Copy link
Contributor Author

Micket commented Aug 22, 2024

Test report by @Micket

Overview of tested easyconfigs (in order)

  • SUCCESS netcdf4-python-1.6.5-foss-2023b.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
vera-skylake-build - Linux Rocky Linux 8.9, x86_64, Intel Xeon Processor (Skylake, IBRS, no TSX), Python 3.6.8
See https://gist.github.com/Micket/fe7c70f7a98bf61dfc899fb8cc2cebb2 for a full test report.

@Micket
Copy link
Contributor Author

Micket commented Aug 22, 2024

Test report by @Micket

Overview of tested easyconfigs (in order)

  • SUCCESS PyZMQ-25.1.2-GCCcore-13.2.0.eb
  • SUCCESS SEPP-4.5.1-foss-2022b.eb
  • SUCCESS sympy-1.12-gfbf-2023b.eb
  • SUCCESS SciPy-bundle-2023.11-gfbf-2023b.eb
  • SUCCESS EggLib-3.3.0-GCC-13.2.0.eb
  • SUCCESS jax-0.4.4-foss-2022a.eb
  • SUCCESS vigra-1.11.2-foss-2023a.eb

Build succeeded for 7 out of 7 (7 easyconfigs in total)
vera-skylake-build - Linux Rocky Linux 8.9, x86_64, Intel Xeon Processor (Skylake, IBRS, no TSX), Python 3.6.8
See https://gist.github.com/Micket/f65845a0504893de88541d44b48432d1 for a full test report.

@Micket
Copy link
Contributor Author

Micket commented Aug 22, 2024

Test report by @Micket

Overview of tested easyconfigs (in order)

  • SUCCESS EasyBuild-4.9.2.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
vera-skylake-build - Linux Rocky Linux 8.9, x86_64, Intel Xeon Processor (Skylake, IBRS, no TSX), Python 3.6.8
See https://gist.github.com/Micket/f9a0187ee654cfafbd899010aaf97132 for a full test report.

@Micket
Copy link
Contributor Author

Micket commented Aug 22, 2024

Test report by @Micket

Overview of tested easyconfigs (in order)

  • SUCCESS TensorRT-8.6.1-foss-2022a-CUDA-11.7.0.eb
  • SUCCESS TensorFlow-2.9.1-foss-2022a.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
vera-skylake-build - Linux Rocky Linux 8.9, x86_64, Intel Xeon Processor (Skylake, IBRS, no TSX), Python 3.6.8
See https://gist.github.com/Micket/d15524a675bdb83242a8c84e2a042c99 for a full test report.

@Micket
Copy link
Contributor Author

Micket commented Aug 22, 2024

Test report by @Micket

Overview of tested easyconfigs (in order)

  • SUCCESS tensorflow-compression-2.11.0-foss-2022a-CUDA-11.7.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis2-01 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz, 1 x NVIDIA Tesla T4, 550.54.14, Python 3.6.8
See https://gist.github.com/Micket/37055a52691ade6faff20ff9cb4e2e0e for a full test report.

@Micket
Copy link
Contributor Author

Micket commented Aug 23, 2024

All the tests above ran fine and produced the correct PYTHONPATH.

@Micket Micket force-pushed the ebpythonprefixes branch 2 times, most recently from a663110 to 4ba7735 Compare September 20, 2024 14:45
@boegel
don't use force_pythonpath + clarify why we're keeping logic on $PYTH…
bc1d2b1 
…ONPATH vs $EBPYTHONPREFIXES in PythonBundle + PythonPackage easyblocks
@boegel boegel mentioned this pull request Sep 30, 2024
Merged
@Micket
Merge pull request #4 from boegel/ebpythonprefixes
64d7ec3 
don't use `force_pythonpath` + clarify why we're keeping logic on `$PYTHONPATH` vs `$EBPYTHONPREFIXES` in `PythonBundle` + `PythonPackage` easyblocks
@boegel boegel changed the title Use prefer_ebpythonprefixes flag to use EBPYTHONPREFIXES update easyblock to let EasyBuild framework take care of prepend to $PYTHONPATH or $EBPYTHONPREFIXES + make PythonBundle and PythonPackage aware of prefer_python_search-path Sep 30, 2024
@boegel boegel changed the title update easyblock to let EasyBuild framework take care of prepend to $PYTHONPATH or $EBPYTHONPREFIXES + make PythonBundle and PythonPackage aware of prefer_python_search-path update easyblock to let EasyBuild framework take care of prepend to $PYTHONPATH or $EBPYTHONPREFIXES + make PythonBundle and PythonPackage aware of --prefer-python-search-path EasyBuild configuration option Sep 30, 2024
@boegel
Copy link
Member

boegel commented Sep 30, 2024

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS LAMMPS-29Aug2024-foss-2023b-kokkos.eb
  • SUCCESS CPLEX-22.1.1-GCCcore-11.2.0.eb
  • SUCCESS AmberTools-23.6-foss-2023a.eb
  • SUCCESS ESMF-8.6.1-foss-2023b.eb
  • SUCCESS Gurobi-9.5.2-GCCcore-11.3.0.eb
  • FAIL (build issue) OpenBabel-3.1.1-gompi-2023a.eb (partial log available at https://gist.github.com/boegel/cd8ffd13dc9b540fc541ad5404882472)
  • SUCCESS OpenCV-4.8.1-foss-2023a-contrib.eb
  • SUCCESS QScintilla-2.11.6-GCCcore-11.2.0.eb
  • SUCCESS Tkinter-3.11.5-GCCcore-13.2.0.eb

Build succeeded for 8 out of 9 (9 easyconfigs in total)
node3518.doduo.os - Linux RHEL 8.8, x86_64, AMD EPYC 7552 48-Core Processor, Python 3.6.8
See https://gist.github.com/boegel/349e7b66f40e34aa3e278c54272a8a49 for a full test report.

@boegel
Copy link
Member

boegel commented Oct 1, 2024

The failure with the OpenBabel tests has nothing to do with the changes in this PR, so I won't let that block this PR...

@boegel
Copy link
Member

boegel commented Oct 1, 2024 •
edited
Loading

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS yaff-1.6.0-foss-2023a.eb
  • SUCCESS matplotlib-3.8.2-gfbf-2023b.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
node3518.doduo.os - Linux RHEL 8.8, x86_64, AMD EPYC 7552 48-Core Processor, Python 3.6.8
See https://gist.github.com/boegel/89b050e16724658b56cfa30bab74a7d1 for a full test report.

edit: using standard configuration, so using $PYTHONPATH in generated module files

@boegel
Copy link
Member

boegel commented Oct 1, 2024 •
edited
Loading

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS yaff-1.6.0-foss-2023a.eb
  • SUCCESS matplotlib-3.8.2-gfbf-2023b.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
node3518.doduo.os - Linux RHEL 8.8, x86_64, AMD EPYC 7552 48-Core Processor, Python 3.6.8
See https://gist.github.com/boegel/acbbd44d4cfb4e9ea085bf9c65f76114 for a full test report.

edit: using --prefer-python-search-path=EBPYTHONPREFIXES

boegel
boegel approved these changes Oct 2, 2024
View reviewed changes
Copy link
Member

@boegel boegel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@boegel boegel merged commit 01c4005 into easybuilders:5.0.x Oct 2, 2024
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@boegel boegel boegel approved these changes

Assignees
No one assigned
Labels
EasyBuild-5.0 EasyBuild 5.0 enhancement
Projects
None yet
Milestone
5.0
Development

Successfully merging this pull request may close these issues.
2 participants
@Micket @boegel