Skip to content

Commit

Permalink
Add _angi to shared directory name when testing HanaSR angi
Browse files Browse the repository at this point in the history
Cluster tests on `svirt` and `pvm_hmc` backends rely on a directory
created on a NFS share, where nodes can share information regarding
node's IP addresses, iSCSI LUNs and other details. This is implemented
in:

https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/tests/ha/setup_hosts_and_luns.pm#L43

This means that if running 2 sets of cluster tests at the same time, on
the same build, same version and same architecture on either of these 2
backends, it can lead to a race condition if both clusters have the same
`CLUSTER_NAME` setting. This is currently happening on the pre-release
job group which has 2 HANA clusters (one testing `SAPHanaSR` resources,
and the other testing `SAPHanaSR-angi` resources) running on
`ppc64le-hmc-sap` workers both with setting `CLUSTER_NAME: hana`.

As can be seen, as the directory name is composed of the cluster name, the
version under test, the architecture and the build, this means that if
tests for 2 clusters with the same name run at the same time for any
given build, both clusters will try to use the same directory to exchange
information introducing a race condition which can lead to failures such
as the one in https://openqa.suse.de/tests/15631415#step/ha_cluster_join/7.
There, the other cluster overwrote the information of the running test,
leading to a failure to add `hana-node01` to `/etc/hosts`.

A first attempt to avoid the race condition was made by replacing the
value of `CLUSTER_NAME: hana` for `CLUSTER_NAME: angi` in the
SAPHanaSR-angi tests in the job group level (MR#801 in openqa_ha_sap
and MR#793 in qam-openqa-yml), however this did not work as there are
several test modules and libraries which rely on `CLUSTER_NAME: hana`
to perform HANA specific actions, such as the takeover. For example:

* tests/ha/check_after_reboot.pm: line 213 (and others)
* tests/ha/fencing.pm: line 70
* lib/hacluster.pm: line 669

So for the time being, cluster tests for hana require the `CLUSTER_NAME`
setting to be set to `hana`.

This commit is a second attempt to avoid the race condition. It adds the
string `_angi` to the shared directory name if running with the
`USE_SAP_HANA_SR_ANGI` setting set to true. This would help us avoid the
existing race condition as the 2 sets of HanaSR tests currently under
test are a classic `SAPHanaSR` and a `SAPHanaSR-angi` one.

Eventually when adding more cluster scenarios on pvm_hmc, further
changes will be required. We will track these in the ticket
https://jira.suse.com/browse/TEAM-9750
  • Loading branch information
alvarocarvajald committed Oct 15, 2024
1 parent 9af8a47 commit 9f3420f
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions tests/ha/setup_hosts_and_luns.pm
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ sub run {
$testname =~ s/@.+$//;
$testname =~ s/_node\d+$//;
$dir_id .= "_$testname" if get_var('HDDVERSION', '');
$dir_id .= '_angi' if (get_var('USE_SAP_HANA_SR_ANGI', '');

set_var('NFS_SUPPORT_DIR', "$mountpt/$dir_id");
assert_script_run "mkdir -p $mountpt";
Expand Down

0 comments on commit 9f3420f

Please sign in to comment.