[2pt] Revise healthcheck predicates

[rosik]

Nowadays there are several different health check predicates in Cartridge.

Eventual failover

cartridge/cartridge/failover.lua

Lines 150 to 161 in e5ca132

	local member = membership.get_member(server.uri)
	if member ~= nil
	and (member.status == 'alive' or member.status == 'suspect')
	and member.payload.uuid == instance_uuid
	and (
	member.payload.state == 'ConfiguringRoles' or
	member.payload.state == 'RolesConfigured'
	) then
	appointments[replicaset_uuid] = instance_uuid
	break
	end

Stateful failover

cartridge/cartridge/roles/coordinator.lua

Lines 34 to 47 in e5ca132

	local server = vars.topology_cfg.servers[instance_uuid]
	if server == nil or not topology.not_disabled(instance_uuid, server) then
	return false
	end
	local member = members[server.uri]
	if member ~= nil
	and (member.status == 'alive' or member.status == 'suspect')
	and (member.payload.uuid == instance_uuid)
	then
	return true
	end
	return false

RPC

cartridge/cartridge/rpc.lua

Lines 41 to 52 in e5ca132

	local function member_is_healthy(uri, instance_uuid)
	local member = membership.get_member(uri)
	return (
	(member ~= nil)
	and (member.status == 'alive')
	and (member.payload.uuid == instance_uuid)
	and (
	member.payload.state == 'ConfiguringRoles' or
	member.payload.state == 'RolesConfigured'
	)
	)
	end

I see two potential problems here:

1. Neither eventual nor stateful failover isn't triggered until a 'suspect' member becomes dead, but RPC already considers it as unhealthy. As a result, get_active_leaders may return a suspect leader and the RPC call would return an error "No remotes with role %q available" preliminary.
2. Stateful failover isn't triggered for OperationError, but eventual failover is. It seems to be slightly inconsistent.

[rosik]

After a brief discussion with @olegrok and @mtrempoltsev we agreed that RPC should consider 'suspects' as healthy.

[rosik]

And I think that OperationError shouldn't be considered unhealthy even if it sounds silly. RPC DoS and additional config applications are even worse taking into account that OperationError is the result of apply_config (or init) failure.

[sharonovd]

Dmitry Sharonov, [15.06.21 14:00]
почему фейловер не сработал, я не понимаю

Dmitry Sharonov, [15.06.21 14:00]
да, мастер гикнулся при накате конфига

Yaroslav Dynnikov, [15.06.21 14:10]
#1139

Да, стейтфул не триггерится на OperationError.