VTOrc: Refactor and reload of ephemeral information for remaining recovery functions

[GuptaManan100]

Description

This is the follow up PR for #10115.

In that PR, we had added the functionality to VTOrc to refresh its ephemeral information before running any cluster operation. But according to our discussions, we should be doing this for all failure scenarios. For example, we see that a tablet has its replication stopped, but we might be using old information and actually it is a BACKUP tablet_type. So, even in these cases, we should not be going ahead with the recovery to start the replication, since it might break other parts of Vitess. This PR addresses that concern and adds this reload of information for the other recovery functions as well.

In this PR, we have also adds some unit tests to VTOrc. We have also refactored the replicationAnalysis codepath to do the locking of the shard and refreshing of the information before we call the recoveryFunction, to prevent repeated code. This PR also deletes any unused failure scenarios that were coming from orchestrator legacy code. Vitess doesn't support Co-primary or intermediate-primary configuration, so it is safe to remove these failure scenarios. We anyways didn't have any associated recoveries for them.

Moreover, the codepath for PrimaryHasPrimary failure has been refactored, to only reset the replication on the primary instance, instead of running a force DeadPrimary recovery.

Related Issue(s)

• RFC - Durability and consensus in Vtorc #8975

Checklist

• "Backport me!" label has been added if this change should be backported
• Tests were added or are not required
• Documentation was added or is not required

Deployment Notes

[GuptaManan100]

This is just a linter fix

[deepthi]

Can you link the catch-all vtorc issue into this PR?
I went and looked at how we decide on PrimaryHasPrimary and now I'm having second thoughts about simply removing the replication. Even though the tablet type is PRIMARY, we should make sure that the underlying mysqld is writable as well. That doesn't seem to be something that is considered when setting IsClusterPrimary to true.

[GuptaManan100]

The PRIMARY tablet not being writable is a failure scenario in itself. If the designated primary is read only, it will spawn the PrimaryIsReadOnly failure which will set the primary to read-write.
I am unsure of what to do about PrimaryHasPrimary too, the failure could be as simple as someone messed up when entering the tablet host port when they wanted to the set replication source (maybe for experimental tablets, etc?). The earlier code assumed that something was seriously wrong for the Primary to have a replication source set, and ran an ERS forcefully.