DAOS-11955 pool: Ensure a PS is inside pool (#13046)

[jolivier23]

• DAOS-11955 pool: Ensure a PS is inside its pool

It was found that a PS leader may enter ds_pool_plan_svc_reconfs with itself being an undesirable replica. This may lead to an assertion failure at "move n replicas from undesired to to_remove" in ds_pool_plan_svc_reconfs. Moreover, such a PS leader may be outside of the pool group, making it incapable of performing many duties that involve collective communication.

This patch therefore ensures that a PS leader will remove undesirable PS replicas synchronously before committing a pool map modification that introduces new undesirable PS replicas. (If we were to keep an undesirable PS replica, it might become a PS leader.)

• Extend and clean up pool_svc_sched. * Allow pool_svc_reconf_ult to return an error, so that we can fail a pool map modification if its synchronous PS replica removal fails. * Allow pool_svc_reconf_ult to get an argument, so that we can tell pool_svc_reconf_ult whether we want a synchronous remove-only run or an asyncrhonous add-remove run. * Move pool_svc_sched.{psc_svc_rf,psc_force_notify} up to pool_svc.
• Prevent pool_svc_step_up_cb from canceling in-progress reconfigurations by comparing pool map versions for which the reconfigurations are scheduled.
• Rename POOL_GROUP_MAP_STATUS to POOL_GROUP_MAP_STATES so that we are consistent with the pool_map module.

Before requesting gatekeeper:

• Two review approvals and any prior change requests have been resolved.
• Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
• Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
• Commit messages follows the guidelines outlined here.
• Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

• You are the appropriate gatekeeper to be landing the patch.
• The PR has 2 reviews by people familiar with the code, including appropriate owners.
• Githooks were used. If not, request that user install them and check copyright dates.
• Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
• All builds have passed. Check non-required builds for any new compiler warnings.
• Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
• If applicable, the PR has addressed any potential version compatibility issues.
• Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
• Extra checks if forced landing is requested
  • Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
  • No new NLT or valgrind warnings. Check the classic view.
  • Quick-build or Quick-functional is not used.
• Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

[github-actions]

Bug-tracker data:
Ticket title is 'pool EMRG src/pool/srv_util.c:477 ds_pool_plan_svc_reconfs() Assertion 'rc != -DER_NONEXIST' '
Status is 'Resolved'
Labels: 'triaged'
https://daosio.atlassian.net/browse/DAOS-11955

[jolivier23]

@liw it wasn't a perfectly clean backport. Can you please take a look and see if anything looks odd?

[daosbuild1]

LGTM. No errors found by checkpatch.

[daosbuild1]

LGTM. No errors found by checkpatch.