Fix: DC lost during wait #1483
Merged
Fix: DC lost during wait #1483
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
freishutz
reviewed
Jul 8, 2024
crmsh/constants.py
Outdated
| @@ -454,4 +454,9 @@ | |||
|
|
|||
| # Commands that are deprecated and hidden from UI | |||
| HIDDEN_COMMANDS = {'ms'} | |||
|
|
|||
| # pacemaker crm/common/results.h | |||
There was a problem hiding this comment.
Just hit another one:
Stderr: �[33mWARNING�[0m: Unknown return code from crmadmin: 113
�[33mWARNING�[0m: DC lost during wait
which is
CRM_EX_UNSATISFIED = 113, //!< Requested item does not satisfy constraints
and it seems like this case mapped on ./lib/common/results.c in pacemaker:
case EAGAIN:
case EBUSY:
return CRM_EX_UNSATISFIED;
So, worth to ignore that RC and give another try.
There was a problem hiding this comment.
So, just reusing the original version of get_dc, ignoring the exit status, is good enough.
freishutz
reviewed
Jul 8, 2024
crmsh/utils.py
Outdated
| if not ServiceManager().service_is_active("pacemaker.service", remote_addr=node): | ||
| raise ValueError("Pacemaker is not running. No DC.") | ||
| dc_deadtime = get_property("dc-deadtime", peer=node) or str(constants.DC_DEADTIME_DEFAULT) | ||
| dc_timeout = int(dc_deadtime.strip('s')) + 5 |
There was a problem hiding this comment.
dc-deadtime value takes time and it could be s|m|h and maybe even d days.
nicholasyang2022
force-pushed
the
refine-pr-1475
branch
3 times, most recently
from
2af3a86 to
cd6f99b
Compare
The initial error that we faced during our online storage upgrade:
```
Command: x Exascaler Install: apply_lustre_params,create_udev_rules,email,emf_agent,emf_node_manager,ha,hosts,ipmi,kdump,logging,lustre,lvm,mdt_backup,modprobe,nics,ntp,os,ost_pools,restart_network,serial,start_cluster,sync_exa_toml (Config ver. 1) failed
User: api
Job: x es-install --steps start_cluster on node5 failed
Step: x Run config-pacemaker on node5 failed (took: 12s 534ms 171us 586ns)
Result (Error):
Bad Exit Code: 1.
Started: 2024-02-07T03:26:16.158Z
Ended: 2024-02-07T03:26:28.692Z
Stdout:
Running Command: config-pacemaker --unmanaged-emf
Stderr:
x Command has failed.
Code: exit status: 1
Stdout: INFO: cib.commit: committed '5e8558de-1ceb-46c2-bd70-1ab4d8504c9f' shadow CIB to the cluster
Stderr: WARNING: DC lost during wait
```
Basically, the source of our problems below (case 3 - DC election or
voiting during cluster recalculation):
```
[root@es-1-virt1 ~]# crmadmin -D -t 1; echo $?
Designated Controller is: es-2-virt1
0
[root@es-1-virt1 ~]# crm cluster stop
INFO: The cluster stack stopped on es-1-virt1
[root@es-1-virt1 ~]# crmadmin -D -t 1; echo $?
error: Could not connect to controller: Connection refused
error: Command failed: Connection refused
102
[root@es-1-virt1 ~]# crm cluster start
INFO: The cluster stack started on es-1-virt1
[root@es-1-virt1 ~]# crmadmin -D -t 1; echo $?
error: No reply received from controller before timeout (1000ms)
error: Command failed: Connection timed out
124
```
Potentially, we have a deadloop in dc_waiter, but it also means that
pacemaker in the same state and in worst case the amount of time should
not be more than 'dc-deadtime'.
nicholasyang2022
force-pushed
the
refine-pr-1475
branch
from
cd6f99b to
1d5ad92
Compare
nicholasyang2022
force-pushed
the
refine-pr-1475
branch
from
1d5ad92 to
2465ff3
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files
Flags with carried forward coverage won't be shown. Click here to find out more.
☔ View full report in Codecov by Sentry. |
freishutz
approved these changes
Jul 16, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a refined version of #1475.
crmsh.utils.wait4dctraces DC transition until it gets stable. However, at certain steps during the transition, there will be no DCs in the cluster. wait4dc does not handle this situation and fails.