Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

let PythonPackage easyblock fix python shebangs by default #3499

Merged
merged 2 commits into from
Dec 4, 2024
Merged

let PythonPackage easyblock fix python shebangs by default #3499

merged 2 commits into from
Dec 4, 2024

Conversation

VRehnberg
Copy link
Contributor

@VRehnberg VRehnberg commented Nov 6, 2024 •
edited
Loading

Fixes #3498

@VRehnberg
Pythonpkg fix_python_shebang_for default ['bin/*']
86946e8 
Make fix_python_shebang_for = ['bin/*'] default in PythonPackage
easyblock. As discussed in:
easybuilders#3498
@VRehnberg VRehnberg mentioned this pull request Nov 6, 2024
Closed
@VRehnberg VRehnberg changed the title PythonPackage fix python shebangs by default, fixes #3498 PythonPackage fix python shebangs by default Nov 6, 2024
@bartoldeman bartoldeman added the EasyBuild-5.0 EasyBuild 5.0 label Nov 20, 2024
@Micket Micket added this to the 5.0 milestone Nov 24, 2024
@VRehnberg
Copy link
Contributor Author

VRehnberg commented Nov 28, 2024 •
edited
Loading

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS accelerate-0.33.0-foss-2023a-CUDA-12.1.1.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-30 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/3486c47db361886fde40ddac959ef5e9 for a full test report.

Manual check of all shebangs. OK

@Micket
Copy link
Contributor

Micket commented Nov 28, 2024

Can you please run a mixed selection of easyconfigs? I'd grab something that uses CMakePythonPackage, PythonPackage, CargoPythonPackage, PythonBundle, just to get a selection of tests and ensure nothing unexpected happens.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS deepdiff-6.7.1-GCCcore-12.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-30 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/345b36451e1bf392c30a74ab6ec3fcd0 for a full test report.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS BeautifulSoup-4.12.2-GCCcore-12.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-14 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/240db6f13957da66e8adff8c90ba3fd2 for a full test report.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS tensorboardX-2.6.2.2-foss-2023a.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-33 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/618640bf0e7d87c7a516f13896af3e64 for a full test report.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS zlib-1.2.12.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-14 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/92f2bced1b90c36e8fbae1e65151a26c for a full test report.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS aiohttp-3.8.5-GCCcore-12.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-14 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/bff7cb237b578950dd70871d1fae3f65 for a full test report.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS binutils-2.39.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-14 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/3a0360ff8cd3589d058e30797e00a820 for a full test report.

@VRehnberg
Copy link
Contributor Author

Can you please run a mixed selection of easyconfigs? I'd grab something that uses CMakePythonPackage, PythonPackage, CargoPythonPackage, PythonBundle, just to get a selection of tests and ensure nothing unexpected happens.

I submitted a few, but was a bit careless so got some more dependencies than I wanted, so will probably spam this PR for a bit. If you have something else you want you can probably build from your own framework_dev with --include-easyblocks-from-pr=3499 -u.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS PyTorch-Lightning-2.2.1-foss-2023a-CUDA-12.1.1.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-14 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/e265364fea77c9375aa23d2284cabf8d for a full test report.

@VRehnberg
Copy link
Contributor Author

VRehnberg commented Nov 28, 2024 •
edited
Loading

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

Build succeeded for 0 out of 1 (1 easyconfigs in total)
alvis3-25 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/ecf3da60db23639180952f6094b1d1bb for a full test report.

ERROR EasyBuild encountered an error (at easybuild/framework/easybuild/base/exceptions.py:126 in __init__): Use of unknown easyconfig parameter 'build_type' when getting parameter value (at easybuild/framework/easybuild/framework/easyconfig/easyconfig.py:1763 in __getitem__)

Probably not related to this pr

@VRehnberg

This comment was marked as off-topic.

Sign in to view
@VRehnberg

This comment was marked as off-topic.

Sign in to view
@VRehnberg

This comment was marked as off-topic.

Sign in to view
@VRehnberg

This comment was marked as off-topic.

Sign in to view
@VRehnberg

This comment was marked as off-topic.

Sign in to view
@VRehnberg

This comment was marked as off-topic.

Sign in to view
@VRehnberg

This comment was marked as off-topic.

Sign in to view
@VRehnberg

This comment was marked as off-topic.

Sign in to view
@VRehnberg

This comment was marked as off-topic.

Sign in to view
@VRehnberg

This comment was marked as off-topic.

Sign in to view
@VRehnberg

This comment was marked as off-topic.

Sign in to view
@VRehnberg

This comment was marked as off-topic.

Sign in to view
@VRehnberg

This comment was marked as off-topic.

Sign in to view
@VRehnberg

This comment was marked as off-topic.

Sign in to view
@VRehnberg

This comment was marked as off-topic.

Sign in to view
@VRehnberg

This comment was marked as off-topic.

Sign in to view
@VRehnberg

This comment was marked as off-topic.

Sign in to view
@VRehnberg

This comment was marked as off-topic.

Sign in to view
@VRehnberg

This comment was marked as off-topic.

Sign in to view
@VRehnberg

This comment was marked as off-topic.

Sign in to view
@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS Pillow-9.4.0-GCCcore-12.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-38 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/ff7c44f6874af0203b945d39a2dafbf3 for a full test report.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS hypothesis-6.68.2-GCCcore-12.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis3-38 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/73cacab1fd399d10f0f67b18c40a1cc0 for a full test report.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS cppy-1.2.1-GCCcore-12.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-37 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/730b2586a1a2454caa098853cceb1321 for a full test report.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS pybind11-2.10.3-GCCcore-12.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-37 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/20e4510b70784b2ebffda3cef6283ba4 for a full test report.

@VRehnberg

This comment was marked as off-topic.

Sign in to view
@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS Tk-8.6.12-GCCcore-12.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-33 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/2ac2e4d96d234c8b44f1377f95b00a94 for a full test report.

@VRehnberg
Copy link
Contributor Author

VRehnberg commented Nov 28, 2024 •
edited
Loading

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

Build succeeded for 0 out of 1 (1 easyconfigs in total)
alvis4-37 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/1411573f48f71c09da8d26b66647b0e7 for a full test report.

Same as SciPy-bundle in another toolchain. buildtype not recognized as an easyconfig parameter. Probably unrelated.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS Tkinter-3.10.8-GCCcore-12.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-33 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/46cec6710dbad393525f4e8a27829e84 for a full test report.

@VRehnberg
Copy link
Contributor Author

VRehnberg commented Nov 28, 2024 •
edited
Loading

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

Build succeeded for 0 out of 1 (1 easyconfigs in total)
alvis4-33 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/78893316d4e0514675c6a892bf6b6970 for a full test report.

This build shouldn't even have started as SciPy-bundle failed.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS t-SNE-CUDA-3.0.1-foss-2023a-CUDA-12.1.1.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-36 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/059331c5461d60c402ea2c932789ab0c for a full test report.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS tokenizers-0.15.2-GCCcore-12.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis3-38 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/421b95c7b82f9a3c49198f788c1b9aef for a full test report.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS cryptography-41.0.1-GCCcore-12.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis3-38 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/7d8e584c3f6d17fd5d8b4562df331204 for a full test report.

@VRehnberg
Copy link
Contributor Author

That should have covered most if not all combinations with some connection to PythonPackage.

There were some consistent failures with scipy in SciPy-bundle about unrecognized easyconfig parameter build_type. I can't see how it could be related, but I can also not see why it appears.

Another unrelated thing is that -x/--extended-dry-run does not seem to work with 5.0.x

== Temporary log file in case of crash /dev/shm/eb-tm46gh6z/easybuild-1b8mye27.log
file /dev/shm/eb-tm46gh6z/fake_vsc_xfdmngdn removed
ERROR: pattern '.*(lmod|LMOD_CMD)' (Lmod) not found in defined 'module' function.
Specify the correct modules tool to avoid weird problems due to this mismatch, see the --modules-tool and --avail-modules-tools command line options.
Or alternatively, use --allow-modules-tool-mismatch to stop treating this as an error. Obtained definition of 'module' function:

@boegel boegel changed the title PythonPackage fix python shebangs by default let PythonPackage easyblock fix python shebangs by default Dec 4, 2024
@boegel
Copy link
Member

boegel commented Dec 4, 2024

That should have covered most if not all combinations with some connection to PythonPackage.

There were some consistent failures with scipy in SciPy-bundle about unrecognized easyconfig parameter build_type. I can't see how it could be related, but I can also not see why it appears.

The problem with build_type was probably introduced via #3454: the custom easyblock for scipy doesn't correctly inherit the extra easyconfig parameters that are defined for MesonNinja, and then the if self.cfg['build_type'] in MesonNinja.configure_step goes sideways.

@VRehnberg Please open an issue in the easyblocks repo on this, maybe @Micket can take a look at that?
Probably worth also checking other easyblocks that derive from MesonNinja...

Another unrelated thing is that -x/--extended-dry-run does not seem to work with 5.0.x

== Temporary log file in case of crash /dev/shm/eb-tm46gh6z/easybuild-1b8mye27.log
file /dev/shm/eb-tm46gh6z/fake_vsc_xfdmngdn removed
ERROR: pattern '.*(lmod|LMOD_CMD)' (Lmod) not found in defined 'module' function.
Specify the correct modules tool to avoid weird problems due to this mismatch, see the --modules-tool and --avail-modules-tools command line options.
Or alternatively, use --allow-modules-tool-mismatch to stop treating this as an error. Obtained definition of 'module' function:

@VRehnberg Please open a framework issue on that, that should definitely be fixed before we push out EasyBuild 5.0.

Make sure you can reproduce it though, because the errors mentioned here point to a very different issue with your session environment, it seems that somehow $LMOD_CMD was not defined (or you weren't using Lmod as modules tool, and EasyBuild wasn't correctly configured to use the alternative Tcl-based environment modules tools).

@boegel boegel added the change label Dec 4, 2024
@VRehnberg VRehnberg mentioned this pull request Dec 4, 2024
Closed
@VRehnberg
Copy link
Contributor Author

VRehnberg commented Dec 4, 2024 •
edited by boegel
Loading

New issues:

boegel
boegel approved these changes Dec 4, 2024
View reviewed changes
Copy link
Member

@boegel boegel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@boegel
Copy link
Member

boegel commented Dec 4, 2024

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS matplotlib-3.8.2-gfbf-2023b.eb
  • SUCCESS tqdm-4.66.5-GCCcore-13.3.0.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
node3501.doduo.os - Linux RHEL 8.8, x86_64, AMD EPYC 7552 48-Core Processor, Python 3.6.8
See https://gist.github.com/boegel/2cdfda3fc55ea42db536323c02f1bc89 for a full test report.

@boegel boegel merged commit 6ea2af3 into easybuilders:5.0.x Dec 4, 2024
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Micket Micket Micket left review comments

@boegel boegel boegel approved these changes

Assignees
No one assigned
Labels
change EasyBuild-5.0 EasyBuild 5.0 enhancement
Projects
None yet
Milestone
5.0
Development

Successfully merging this pull request may close these issues.
4 participants
@VRehnberg @Micket @boegel @bartoldeman