Skip to content

Commit

Permalink
Merge branch 'develop'
Browse files Browse the repository at this point in the history
  • Loading branch information
boegel committed Sep 11, 2023
2 parents dbc4f86 + 03891cb commit 76e5e33
Show file tree
Hide file tree
Showing 13 changed files with 1,926 additions and 466 deletions.
3 changes: 3 additions & 0 deletions docs/changelog-docs.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ search:

(for EasyBuild release notes, see [EasyBuild release notes][release_notes])

- **release 20230911.01** (*9 Sept 2023*): update
release notes for EasyBuild v4.8.1 (see
[EasyBuild v4.8.1 (11 Sept 2023)][release_notes_eb481])
- **release 20230707.01** (*7 July 2023*): update
release notes for EasyBuild v4.8.0 (see
[EasyBuild v4.8.0 (7 July 2023)][release_notes_eb480])
Expand Down
2 changes: 1 addition & 1 deletion docs/code-style.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ The only (major) exception to PEP8 is our preference for longer line lengths: li
## Notes

Code style in easyconfig files can be **automatically checked** using `--check-contrib`,
for example: `eb --check-contrib sympy-1.3-intel-2018a-Python-2.7.14.eb`
for example: `eb --check-contrib HPL-2.3-foss-2022b.eb`
(see [Code style review][contributing_review_process_code_style] for more details).

Style guides that go a step beyond PEP8:
Expand Down
129 changes: 105 additions & 24 deletions docs/hooks.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,22 +43,28 @@ eb ...

## Available hooks

Currently (since EasyBuild v3.7.0), three types of hooks are supported:
Since EasyBuild v4.8.1, six different types of hooks are supported:

* `start_hook` and `end_hook`, which are triggered *once* before starting software installations,
and *once* right after completing all installations, respectively
* `start_hook`, `pre_build_and_install_loop_hook`, `post_build_and_install_loop_hook`, and `end_hook` which are triggered *once* right after
EasyBuild starts, *once* before looping over the easyconfigs to be built, *once* after completing the loop over the eayconfigs to be installed,
and *once* shortly before EasyBuild completes, respectively.
* `parse_hook`, which is triggered when an easyconfig file is being parsed
* `module_write_hook`, which is triggered right before a module file is written.
This includes the temporary module file used when installing extensions and during the sanity check,
as well as the devel module.
* "*step*" hooks that are triggered before and after every step of each installation procedure that is performed,
also aptly named '`pre`'- and '`post`'-hooks
also aptly named '`pre`'- and '`post`' hooks.
* `cancel_hook` and `fail_hook` which are triggered when a `KeyboardInterrupt` or `EasyBuildError` is raised,
respectively.
* `pre_run_shell_cmd_hook` and `post_run_shell_cmd_hook` which are triggered right before and after running
a shell command, respectively.

The list of currently available hooks in order of execution,
which can also be consulted using `eb --avail-hooks`, is:

* `start_hook` *(only called once in an EasyBuild session)*
* `parse_hook` *(available since EasyBuild v3.7.0)*
* `pre_build_and_install_loop` *(available since EasyBuild v4.8.1)*
* `pre_fetch_hook`, `post_fetch_hook`
* `pre_ready_hook`, `post_ready_hook`
* `pre_source_hook`, `post_source_hook`
Expand All @@ -68,16 +74,27 @@ which can also be consulted using `eb --avail-hooks`, is:
* `pre_build_hook`, `post_build_hook`
* `pre_test_hook`, `post_test_hook`
* `pre_install_hook`, `post_install_hook`
* `pre_extensions_hook`, `post_extensions_hook`
* `pre_extensions_hook`
* `pre_single_extension_hook`, `post_single_extension_hook` *(available since EasyBuild v4.7.1)*
* `post_extensions_hook`
* `pre_postiter_hook`, `post_postiter_hook` *(available since EasyBuild v4.8.1)*
* `pre_postproc_hook`, `post_postproc_hook`
* `pre_sanitycheck_hook`, `post_sanitycheck_hook`
* `pre_cleanup_hook`, `post_cleanup_hook`
* `pre_module_hook`, `post_module_hook`
* `pre_module_hook`
* `module_write_hook` *(called multiple times per installation, available since EasyBuild v4.4.1)*
* `post_module_hook`
* `pre_permissions_hook`, `post_permissions_hook`
* `pre_package_hook`, `post_package_hook`
* `pre_testcases_hook`, `post_testcases_hook`
* `post_build_and_install_loop` *(available since EasyBuild v4.8.1)*
* `end_hook` *(only called once in an EasyBuild session)*
* `module_write_hook` *(called multiple times per installation, available since EasyBuild v4.4.1)*
* `cancel_hook` *(available since EasyBuild v4.8.1)*
* `fail_hook` *(available since EasyBuild v4.8.1)*

`pre_run_shell_cmd_hook` and `post_run_shell_cmd_hook` *(available since EasyBuild v4.8.1)* are not included in
the list above because they can not be put in a particular order relative to other hooks, since these hooks
are triggered several times throughout an EasyBuild session.

All functions implemented in the provided Python module for which the name ends with `_hook` are considered.

Expand All @@ -102,20 +119,46 @@ Run 'eb --avail-hooks' to get an overview of known hooks
To implement hooks, simply define one or more functions in a Python module (`*.py`),
each named after an available hook.

In hooks you have access to the full functionality provided by the EasyBuild framework,
so do `import` from `easybuild.tools.*` (or other `easybuild.*` namespaces) to leverage
those functions.

Do take into account the following:

* for `start_hook` and `end_hook`, no arguments are provided
* for `parse_hook`, one argument is provided: the `EasyConfig` instance
that corresponds to the easyconfig file being parsed (usually referred to as `ec`)
* for `module_write_hook`, 3 arguments are provided:
* the `EasyBlock` instance used to perform the installation (usually referred to as `self`)
* the filepath of the module that will be written
* the module text as a string
The return value of this hook, when set, will replace the original text that is then written to the module file.
* for the step hooks, one argument is provided:
the `EasyBlock` instance used to perform the installation (usually referred to as `self`)
* the parsed easyconfig file can be accessed in the step hooks via the `EasyBlock` instance,
i.e., via `self.cfg`
* [Hook arguments][hooks-arguments]
* [Return value of hooks][hooks-return-value]
* [Parse hook specifics][parse-hook-specifics]


### Hook arguments {: #hooks-arguments }

* For both `start_hook` and `end_hook` no arguments are provided.
* For `cancel_hook` and `fail_hook` the `KeyboardInterrupt` or `EasyBuildError` exception that was raised
is provided as an argument.
* For `parse_hook` the `EasyConfig` instance that corresponds to the easyconfig file being parsed
(usually referred to as `ec`) is passed as an argument.
* For `pre_build_and_install_loop_hook` a list of easyconfigs is provided as an argument.
* For `post_build_and_install_loop_hook` a list of easyconfigs with build results is provided as an argument.
* For `pre_run_shell_cmd_hook`, multiple arguments are passed:
* An unnamed argument (often called `cmd`) that corresponds to the shell command that will be run,
which could be provided either as a string value (like `"echo hello"`) or a list value (like `['echo', 'hello']`).
* A named argument `work_dir` that specifies the path of the working directory in which the command will be run.
* For interactive commands (which are run via the `run_cmd_qa` function), an additional named argument
`interactive` is set to `True`.
* For `post_run_shell_cmd_hook`, multiple arguments are passed:
* An unnamed argument (often called `cmd`) that corresponds to the shell command that was run,
which could be provided either as a string value (like `"echo hello"`) or a list value (like `['echo', 'hello']`).
* A named argument `work_dir` that specifies the working directory in which the shell command was run.
* A named argument `exit_code` that specifies the exit code of the shell command that was run.
* A named argument `output` that specifies the output of the shell command that was run.
* For interactive commands (which are run via the `run_cmd_qa` function), an additional named argument
`interactive` is set to `True`.
* For `module_write_hook`, 3 arguments are provided:
* The `EasyBlock` instance used to perform the installation (usually referred to as `self`).
* The filepath of the module that will be written.
* The module text as a string.
* For the step hooks, the `EasyBlock` instance used to perform the installation (usually referred to as `self`).
The parsed easyconfig file can be accessed in the step hooks via the `EasyBlock` instance, i.e., via `self.cfg`.

It is recommended to anticipate possible changes in the provided (named) arguments,
using the `*args` and `**kwargs` mechanism commonly used in Python. This
Expand All @@ -127,11 +170,22 @@ def pre_configure_hook(self, *args, **kwargs):
...
```

In hooks you have access to the full functionality provided by the EasyBuild framework,
so do `import` from `easybuild.tools.*` (or other `easybuild.*` namespaces) to leverage
those functions.

### Parse hook specifics
### Return value of hooks {: #hooks-return-value }

The return value of a hook is usually ignored by EasyBuild, except in particular cases:

* If the `module_write_hook` returns a (string) value, it **replaces the original text that was going to be
written to the module file**. This way the `module_write_hook` can extend, change, or entirely replace the
module text that was provided as an argument.

* If the `pre_run_shell_cmd_hook` returns a value, it **replaces the shell command that was going to be run**.
Hence, this hook can change or entirely replace particular shell commands right before they are executed.
Note that the value type of the return value of `pre_run_shell_cmd_hook` *must* match with the type of the
first (unnamed) argument that provides the shell command that would have been run originally.


### Parse hook specifics {: #parse-hook-specifics }

`parse_hook` is triggered right *after* reading the easyconfig file,
before further parsing of some easyconfig parameters (like `*dependencies`) into
Expand Down Expand Up @@ -220,7 +274,7 @@ To achieve the intended effect, you can either:
```

A better approach for manipulating easyconfig parameters is to use the `parse_hook` that
was introduced in EasyBuild v3.7.0 (see [Parse hook specifics](#parse-hook-specifics)),
was introduced in EasyBuild v3.7.0 (see [Parse hook specifics][parse-hook-specifics]),
where these kind of surprises will not occur (because templating is automatically disabled
before `parse_hook` is called and restored immediately afterwards).
See also [Example hook to inject a custom patch file](#inject-a-custom-patch-file).
Expand Down Expand Up @@ -276,3 +330,30 @@ def module_write_hook(self, filepath, module_txt, *args, **kwargs):
replace = 'prepend_path("EBPYTHONPREFIXES", root)'
return re.sub(search, replace, module_txt)
```

### Log running of shell commands + prepend `make install` with `sudo`

```py
shell_cmds_log = '/tmp/eb_shell_cmds.log'

def pre_run_shell_cmd_hook(cmd, work_dir=None, interactive=None):
"""
Log shell commands before they are run,
and replace 'make install' with 'sudo make install'.
"""
with open(shell_cmds_log, 'a') as fp:
cmd_type = 'interactive' if interactive else 'non-interactive'
fp.write("%s command '%s' will be run in %s ...\n" % (cmd_type, cmd, work_dir))

if cmd == "make install":
return "sudo make install"


def post_run_shell_cmd_hook(cmd, work_dir=None, interactive=None, exit_code=None, output=None):
"""
Log shell commands that were run.
"""
with open(shell_cmds_log, 'a') as fp:
cmd_type = 'interactive' if interactive else 'non-interactive'
fp.write("%s command '%s' in %s exited with %s - output: %s\n" % (cmd_type, cmd, work_dir, exit_code, output))
```
76 changes: 38 additions & 38 deletions docs/partial-installations.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,18 +17,18 @@ The following step names are recognized (listed in execution order): `fetch`, `r
Example usage:

``` console
$ eb GCC-4.9.2.eb --stop configure
$ eb GCC-11.3.0.eb --stop configure
== temporary log file in case of crash /tmp/eb-X2Z0b7/easybuild-mGxmNb.log
== processing EasyBuild easyconfig /home/example/GCC-4.9.2.eb
== building and installing GCC/4.9.2...
== processing EasyBuild easyconfig /home/example/GCC-11.3.0.eb
== building and installing GCC/11.3.0...
== fetching files...
== creating build dir, resetting environment...
== unpacking...
== patching...
== preparing...
== configuring...
== COMPLETED: Installation STOPPED successfully
== Results of the build can be found in the log file /dev/shm/example/GCC/4.9.2/dummy-dummy/easybuild/easybuild-GCC-4.9.2-20150430.091644.log
== Results of the build can be found in the log file /dev/shm/example/GCC/11.3.0/dummy-dummy/easybuild/easybuild-GCC-11.3.0-20230613.091644.log
== Build succeeded for 1 out of 1
== temporary log file(s) /tmp/eb-X2Z0b7/easybuild-mGxmNb.log* have been removed.
== temporary directory /tmp/eb-X2Z0b7 has been removed.
Expand All @@ -44,15 +44,15 @@ Example usage:

``` console
$ eb GCCcore-6.2.0.eb --fetch
== temporary log file in case of crash /tmp/eb-1ZZX2b/easybuild-NSmm5P.log
== processing EasyBuild easyconfig /home/example/GCCcore-6.2.0.eb
== building and installing GCCcore/6.2.0...
== Temporary log file in case of crash /tmp/eb-23iu2fpm/easybuild-ss0wcidi.log
== processing EasyBuild easyconfig /home/example/GCCcore-11.3.0.eb
== building and installing GCCcore/11.3.0...
== fetching files...
== COMPLETED: Installation STOPPED successfully
== Results of the build can be found in the log file(s) /dev/shm/example/GCC/4.9.2/dummy-dummy/easybuild/easybuild-GCCcore-6.2.0-20180330.170523.log
== COMPLETED: Installation STOPPED successfully (took 1 secs)
== Results of the build can be found in the log file(s) /tmp/example/easybuild/build/GCCcore/11.3.0/system-system/easybuild/easybuild-GCCcore-11.3.0-20230613.090338.log
== Build succeeded for 1 out of 1
== Temporary log file(s) /tmp/eb-1ZZX2b/easybuild-NSmm5P.log* have been removed.
== Temporary directory /tmp/eb-1ZZX2b has been removed.
== Temporary log file(s) /tmp/example/eb-23iu2fpm/easybuild-ss0wcidi.log* have been removed.
== Temporary directory /tmp/example/eb-23iu2fpm has been removed.
```

!!! note
Expand Down Expand Up @@ -87,17 +87,17 @@ vi) regenerate the module file (since it contains a list of installed extensions
Example usage:

``` console
$ eb Python-2.7.9-intel-2015a.eb --skip
$ eb Python-3.10.4-GCCcore-11.3.0.eb --skip
...
== Python/2.7.9-intel-2015a is already installed (module found), skipping
== Python/3.10.4-GCCcore-11.3.0 is already installed (module found), skipping
== No easyconfigs left to be built.
== Build succeeded for 0 out of 0
```

``` console
$ eb Python-2.7.9-intel-2015a.eb --skip --rebuild
$ eb Python-3.10.4-GCCcore-11.3.0.eb --skip --rebuild
...
== building and installing Python/2.7.9-intel-2015a...
== building and installing Python/3.10.4-GCCcore-11.3.0...
...
== configuring [skipped]
== building [skipped]
Expand Down Expand Up @@ -168,11 +168,11 @@ Example usage:
``` console
$ module avail GCC
---------------------------- /home/example/.local/modules/all ----------------------------
GCC/4.8.2
GCC/11.2.0
$ eb GCC-5.1.0.eb --module-only --rebuild
$ eb GCC-11.3.0.eb --module-only --rebuild
...
== building and installing GCC/5.1.0...
== building and installing GCC/11.3.0...
== fetching files [skipped]
...
== configuring [skipped]
Expand All @@ -189,31 +189,31 @@ Example usage:
$ module avail GCC

---------------------------- /home/example/.local/modules/all ----------------------------
GCC/4.8.2 GCC/5.1.0
GCC/11.2.0 GCC/11.3.0
```

* regenerate existing module file:

``` console
$ module avail GCC/4.8.2
$ module avail GCC/11.3.0

---------------------------- /home/example/.local/modules/all ----------------------------
GCC/4.8.2
GCC/11.3.0

$ ls -l /home/example/.local/modules/all/GCC/4.8.2
-rw-rw-r-- 1 example example 1002 Jan 11 17:19 /home/example/.local/modules/all/GCC/4.8.2
$ ls -l /home/example/.local/modules/all/GCC/11.3.0
-rw-rw-r-- 1 example example 1002 Jun 13 08:35 /home/example/.local/modules/all/GCC/11.3.0

$ eb GCC-4.8.2.eb --module-only --rebuild
$ eb GCC-11.3.0.eb --module-only --rebuild
...
== building and installing GCC/4.8.2...
== building and installing GCC/11.3.0...
...
== sanity checking [skipped]
== creating module...
== COMPLETED: Installation ended successfully
...

$ ls -l /home/example/.local/modules/all/GCC/4.8.2
-rw-rw-r-- 1 example example 1064 Apr 30 10:54 /home/example/.local/modules/all/GCC/4.8.2
$ ls -l /home/example/.local/modules/all/GCC/11.3.0
-rw-rw-r-- 1 example example 1064 Jun 13 10:54 /home/example/.local/modules/all/GCC/11.3.0
```


Expand All @@ -228,26 +228,26 @@ Examples:
`--module-only --module-syntax=Lua` can be used:

``` console
$ module avail GCC/4.8.2
$ module avail GCC/11.3.0

---------------------------- /home/example/.local/modules/all ----------------------------
GCC/4.8.2
GCC/11.3.0

$ ls -l /home/example/.local/modules/all/GCC/4.8.2*
-rw-rw-r-- 1 example example 1064 Apr 30 10:54 /home/example/.local/modules/all/GCC/4.8.2
$ ls -l /home/example/.local/modules/all/GCC/11.3.0*
-rw-rw-r-- 1 example example 1064 Jun 13 08:54 /home/example/.local/modules/all/GCC/11.3.0

$ eb GCC-4.8.2.eb --modules-tool=Lmod --module-only --module-syntax=Lua --rebuild
$ eb GCC-11.3.0.eb --modules-tool=Lmod --module-only --module-syntax=Lua --rebuild
...
== building and installing GCC/4.8.2...
== building and installing GCC/11.3.0...
...
== sanity checking [skipped]
== creating module...
== COMPLETED: Installation ended successfully
...

$ ls -l /home/example/.local/modules/all/GCC/4.8.2*
-rw-rw-r-- 1 example example 1064 Apr 30 10:54 /home/example/.local/modules/all/GCC/4.8.2
-rw-rw-r-- 1 example example 1249 Apr 30 11:56 /home/example/.local/modules/all/GCC/4.8.2.lua
$ ls -l /home/example/.local/modules/all/GCC/11.3.0*
-rw-rw-r-- 1 example example 1064 Jun 13 08:54 /home/example/.local/modules/all/GCC/11.3.0
-rw-rw-r-- 1 example example 1249 Jun 13 08:56 /home/example/.local/modules/all/GCC/11.3.0.lua
```

!!! note
Expand All @@ -263,7 +263,7 @@ Examples:
``` console
$ eb --installpath-modules=$HOME/test/modules --module-only --module-naming-scheme=HierarchicalMNS --rebuild
...
== building and installing Core/GCC/4.8.2...
== building and installing Core/GCC/11.3.0...
...
== sanity checking [skipped]
== creating module...
Expand All @@ -274,7 +274,7 @@ Examples:
$ module avail

---------------------------- /home/example/test/modules/all ----------------------------
Core/GCC/4.8.2
Core/GCC/11.3.0
```

!!! note
Expand Down
4 changes: 2 additions & 2 deletions docs/python-2-3-compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ compatible with both Python versions 2 and 3. More specifically, the
following Python versions are currently supported:

- Python 2.6.x (support was removed in EasyBuild v4.4.0)
- Python 2.7.x
- Python 3.5.x
- Python 2.7.x ([support will be removed in EasyBuild v5.0.0](easybuild-v5/overview-of-changes.md))
- Python 3.5.x ([support will be removed in EasyBuild v5.0.0](easybuild-v5/overview-of-changes.md))
- Python 3.6.x
- Python 3.7.x
- Python 3.8.x (requires EasyBuild v4.1.0)
Expand Down
Loading

0 comments on commit 76e5e33

Please sign in to comment.