Skip to content

Conference call notes 20241106

Jump to bottom
Kenneth Hoste edited this page Nov 6, 2024 · 1 revision

(back to Conference calls)

Notes on the 258th EasyBuild conference call, Wednesday 6 Nov 2024 (09:00 UTC / 10:00 CET)

Attendees

List of attendees (15):

  • David Coster (Max Planck Institute for Plasma Physics, Germany)
  • Alex Domingo (Vrije Universiteit Brussel, Belgium)
  • Davide Grassano (CECAM, Switzerland)
  • Jasper Grimm (University of York, UK)
  • Leonardo Honfi Camilo (Univ. of Wageningen, Netherlands)
  • Kenneth Hoste (HPC-UGent, Belgium)
  • Adam Huffman (Big Data Institute, Oxford, UK)
  • Diana Iusan (SNIC/UPPMAX, Sweden)
  • Kurt Lust (UAntwerpen, Belgium + LUMI User Support Team)
  • Alan O'Cais (CECAM, Univ. of Barcelona)
  • Lara Peeters (HPC-UGent, Belgium)
  • Jan Reuter (JSC, Germany)
  • Jörg Saßmannshausen (Imperial College London, UK)
  • Alain van Hoof (TU Eindhoven, Netherlands)
  • Cintia Willemyns (Vrije Universiteit Brussel, Belgium)

Agenda

  • overview of recent developments
  • outlook to EasyBuild 5.0 release
  • 2024a/2024b update of common toolchains
  • Q&A

Recent developments

  • latest EasyBuild release: 4.9.4 (22 Sept 2024)
  • next EasyBuild release:
    • very likely EasyBuild v5.0.0 🔥
    • additional EasyBuild 4.9.x versions could still be done via 4.9.x branches, but none are planned currently
  • EasyBuild v5.0.0
    • project board: https://github.com/orgs/easybuilders/projects/18/views/2
    • detailed notes on latest developments in https://hackmd.io/tZtz6vMmRfGL6N9Rf__OEA
    • to test development version of EasyBuild 5.0: 
      # set up Python virtual environment, and jump into it
python3 -m venv eb5
source eb5/bin/activate

# install EasyBuild 5.0 development version into it
pip install https://github.com/easybuilders/easybuild-framework/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyblocks/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyconfigs/archive/5.0.x.tar.gz

# go!
eb --version

Merged PRs

  • docs (merged PRs)

    • ...

  • framework (merged PRs)

    • bug fixes
      • [5.0.x] Reverse order for parsing files in $XDG_CONFIG_DIRS (PR #4630)
    • enhancements
      • [develop] updates and fixes for findPythonDeps script (PR #4682)
    • changes
      • ...
    • code cleanup
      • ...

  • easyblocks (merged PRs)

    • bug fixes
      • [develop] fix setting of CMAKE_OPTIONS for SuiteSparse versions between 5.1.2 and 6.0.0 (PR #3471)
      • [develop] fix check in Score-P's configure scripts that may fail if the path to certain dependencies include yes or no (PR #3496)
    • enhancements
      • ...
    • updates
      • [develop] Fix oneAPI sanity check for ifort removal in 2025.0 and newer (PR #3495)
    • changes
      • ...
    • new easyblocks
      • ...
    • code cleanup
      • ...

  • easyconfigs (merged PRs)

    • ~50 easyconfig PRs were merged since last conf call
    • bug fixes/reports
      • [develop] Fix fastparquet v2023.4.0 using CargoPythonBundle easyblock (PR #20775)
      • [develop] add patch to SciPy-bundle 2024.05 that fixes test failure on aarch64 (PR #21559)
      • [develop] fix CI check for extension patches using alt_location (PR #21700)
      • [develop] remove -m64 compiler flag for non-x86-64 CPU architectures in SIONlib 1.7.7 easyconfigs (PR #21752)
      • [develop] add missing perl-bundle-cpan to recent parallel versions (PR #21776)
      • [develop] use new checksum for source tarball of SIONlib 1.7.7 (PR #21748)
    • enhancements
      • [develop] Add a github workflow that tags PRs with new/update/change + add diffs in comments (PR #21758)
    • (noteworthy) new software
      • ...
    • noteworthy software updates
      • ...
    • cleanup
      • ...
    • changes
      • ...

Open (active) PRs

  • docs (open PRs + issues)

    • ...

  • framework (open PRs + issues)

    • bug fixes
      • [5.0.x] improve portability of reproducible tarballs by replacing external tar command with tarfile module (PR #4660)
      • [develop] fix dry-run output when using multi_deps (PR #4678)
      • [5.0.x] make LooseVersion('1.0') == LooseVersion('1') (PR #4691)
    • enhancements
      • [develop] Copy build log and artifacts to a permanent location after failures (WIP PR #4601)
      • [develop] allow extensions to add text to the module file (PR #4652)
      • Problem using $CPATH in modulefiles overwriting system paths (issue #3331)
        • [5.0.x] add --search-path-cpp-headers configuration option to control how EasyBuild sets paths to headers at build time (PR #4645)
        • [5.0.x] refactor generation of required environment variables in module files (PR #4653)
          • test was added to verify that making ModuleLoadEnvironment a singleton causes trouble
          • needs more love, additional test is broken
          • easyblocks will need to be updated accordingly to not use make_module_req_guess anymore (since it was deprecated)
        • [5.0.x] add module-search-path-headers configuration option to control how modules set search paths to header files (PR #4655)
      • [develop] add support for injecting checksums for cargo crates (PR #4661)
      • [5.0.x] Add function to modify path-like environment variable in a context (PR #4681)
      • [5.0.x] Allow use of custom delimiter for paths in module generator (PR #4687)
      • [5.0.x] Add global option to set default toolchain option of "debug" (PR #4688)
      • [5.0.x] add a CUDA device code sanity check (PR #4692)
      • [5.0.x] add support for developer mode (WIP PR #4693)
        • allows to specify a particular prepared unpacked source directory rather than unpacking source tarball
    • code cleanup
      • [5.0.x] simplify code for determining the $PYTHONPATH module entries (PR #4686)
      • [5.0.x] simplify loop in collect_exts_file_info (PR #4689)
    • changes
      • [5.0.x] Rename SOURCE_STEP to EXTRACT_STEP (PR #4629)
      • [5.0.x] Deprecate use of parallel easyconfig parameter and fix updating the template value (PR #4580)
      • [5.0.x] Deprecate post_install_step, rename it to post_processing_step (issue #4656)
      • [5.0.x] proposed changest to add a timeout to run_shell_cmd (PR #4665)
      • [develop] Added possibility to call amend/try-amend multiple times from easystack (PR #4667)
      • [5.0.x] Let jobs retweak easyconfigs themselves (draft PR #4669)
      • [5.0.x] mark support for installing extensions in parallel as being mature, since it's no longer experimental (PR #4672)
      • [5.0.x] mark easystack support as being mature, since it's no longer experimental (PR #4673)
      • [5.0.x] Refactor make_extension_string (PR #4690)

  • easyblocks (open PRs + issues)

    • bug fixes
      • [5.0.x] run pip check only once for PythonBundle (PR #3432)
      • [develop] Avoid failure in Python sanity check when PIP_REQUIRE_VIRTUALENV is set (PR #3460)
      • [develop] take into account that f90wrap_version may be None in MetalWalls easyblock (PR #3468)
      • [develop] fix setting of CMAKE_OPTIONS for SuiteSparse versions between 5.1.2 and 6.0.0 (PR #3471)
      • ORCA Easyblock does not install packaged executable in ORCA 6.0.0 (issue #3479)
      • [develop] tweak libpaths in TensorFlow easyblock by adding directory containing libnccl.so.2 (PR #3497)
    • enhancements
      • [develop] ignore user python packages by using -s option when running Python code via "python -c" (PR #3440)
      • [develop] enhance generic Bundle EasyBlock to transfer module requirements of components (PR #3472)
    • updates
      • [develop] update custom easyblock for CP2K >= v2024 (PR #3433)
    • changes
      • Put our Python sitecustomize.py into the site-packages folder (issue #3493)
      • Should fix_python_shebang_for="bin/*" be default in PythonPackage? (issue #3498)
    • new
      • [develop] add custom easyblock for Bandicoot (PR #3380)
      • [develop] custom easyblock for DeepSpeed (PR #3450)

  • easyconfigs (open PRs + issues)

    • bug fixes/reports
      • [develop -> 5.0.x] switch to CargoPythonBundle easyblock for orjson 3.9.15 (PR #21392)
        • requires being able to produce reproducible source tarballs from Git repository (WIP in 5.0.x branch of EasyBuild framework)
      • [5.0.x] promote pybind11 to runtime dependency for CPPE (to fix pip check failure) (PR #21564)
    • enhancements
      • [develop] ignore user-installed packages when running python in sanitycheck (PR #21362)
      • [develop] also install w90spn2spn.x + w90chk2chk.x for Wannier90 3.1.0 (PR #21383)
      • [develop] Change tagbot workflow to run on pull_request_target and comment on security (PR #21779)
    • (noteworthy) new software
      • ...
    • software updates
      • [develop] PyTorch v2.3.0 (PR #20489)
        • blocked by changes in how results of PyTorch test suite are reported
    • changes
      • [5.0.x] Replace parallel by maxparallel (PR #19375)
      • [develop] GTDB-Tk v2.4.0 should use skani and not FastANI (PR #21518)

2024b common toolchains

  • GCC 14.2 as a base (see easyconfigs PR #21114)
  • we should define candidate toolchains using latest version of all components
    • are newer versions of Intel components compared to intel/2024a already available?
    • 2025.0 version removes ifort (classic Fortran compiler), replaced with ifx (oneAPI Fortran compiler)
  • should we consider replacing intel with Clang-based toolchain as 2nd common toolchain next to foss?
    • Fortran frontend in Clang is becoming mature enough?

Q&A / others

  • EUM'25 will be 25-27 March 2025 at Jülich Supercomputing Centre
  • (Jörg) feedback welcome on PRs for VASP
  • (Jörg) feedback welcome on Molcas PRs: PR #21764 + PR #21780
  • (Alex) errors/warnings when doing import tensorflow for TensorFlow 2.15 w/ CUDA 12.1
    • are others seeing warnings too related to cuDNN? 
      Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
    • Alex will open an issue with more details, see also upstream: https://github.com/tensorflow/tensorflow/issues/62075
Clone this wiki locally