Analysis: locked semi sync master

[shlomi-noach]

Replaces #1173
A different approach to detecting LockedSemiSyncMaster

[shlomi-noach]

This PR introduces the LockedSemiSyncMaster analysis:

• When a master has semi-sync enabled
• And when it sees insufficient semi-sync replicas (Rpl_semi_sync_master_clients < rpl_semi_sync_master_wait_for_slave_count)
• And it does not fall back to async (rpl_semi_sync_master_timeout is high)
• And more than config.Config.ReasonableReplicationLagSeconds passes

then the master is considered to be LockedSemiSyncMaster: it cannot accept writes, and that's a failure detection scenario.

When first three conditions are met, and 4th is not met yet, the analysis is LockedSemiSyncMasterHypothesis. While it shows on replication-analysis command, it is non-actionable, and merely serves as preparation step for LockedSemiSyncMaster.

Analyzing a LockedSemiSyncMaster scenario is different than all other current analysis, because it takes time to make the analysis. MySQL does not provide the mechanism to say "this master has been locked for the past 15 seconds". orchestrator needs to figure this out reliably.

Right now this PR offers no failover hooks for this analysis.

[earl86]

Can it auto disable semi-sync on master, when orchestrator find LockedSemiSyncMaster?

[shlomi-noach]

Can it auto disable semi-sync on master, when orchestrator find LockedSemiSyncMaster?

As mentioned above, right now this PR offers no failover hooks for this analysis. Followup work will offer taking action, and this is still in the works. Action might be:

• invoke special hooks?
• disable semi-sync on master?
• enable semi-sync on another replica?
• failover master AND enable semi sync on another replica? (avoid split brain scenario)

There can be many rules: e.g. only enable semi sync in a replica in same dc/same region/different dc/different region ; only provision up the number of semi-sync replicas / provision up and down to match exact expectation / etc...