Fix rpc.get_candidates function

[hackallcode]

local rpc = require('cartridge.rpc')
local _, err = rpc.call('my_role', ...)
local candidates = rpc.get_candidates('my_role')
log.info(err)
log.info(candidates)

Sometimes the code above may log ["localhost:11002"] and RemoteCallError: "localhost:11002": Role "my_role" unavailable simultaneously. It means that rpc.get_candidates returns those instances that are not actually alive, i.e. nothing can be done on them. So, it's incorrect behavior of rpc.get_candidates.

This seems to be due to the slow writing of the configuration backup to disk. As a result, we may have such that all instances are in the RolesConfigured status, but the config has not yet begun to be applied to some of the instances (in the example it's localhost:11002).

To reproduce this error, you can:

• create 2 instances (with 11001 and 11002 ports);

• assign my_role only to the second instance;

• insert code:

  if confapplier.get_advertise_uri() == 'localhost:11002' then
      local fiber = require('fiber')
      fiber.sleep(3)
  end

  to the 135 line in cartridge/twophase.lua file:

  cartridge/cartridge/twophase.lua

  Lines 130 to 136 in e887629

  	local function commit_2pc()
  	Commit2pcError:assert(
  	vars.prepared_config ~= nil,
  	"commit isn't prepared"
  	)
  	local workdir = confapplier.get_workdir()

[Steap2448]

The original problem was get_candidates (called on A1) returning instances not suitable for requested role (B1). For example it occurs when twophase on B1 is stuck on commit phase and new role has not been applied on it. A1 has no idea that B1 hasn't applied new config and rightfully thinks that B1 is a good candidate for a role, that was previously assigned to it, judging from its local topology. The problem is in different applied configs.

Currently there is no reliable way to know which config is applied on different replicas. All solutions boil down to increasing chances of success. But there is no guarantee.

@olegrok suggests to use topology from applied configuration and specify uri manually. See #1588 for details.

Also consider usage of retries and be vigilant.

[artur-barsegyan]

What is the reason for that triage?

[yngvar-antonsson]

You can't rely on a result of rpc.get_candidates. After a get_candidates call you can lose one of the candidates. Now we can't fix this issue, so I close it.
To avoid this bug you can:

• use stateful failover to make your cluster more reliable
• implement @Steap2448 patch
• get rid of rpc module calls

[hackallcode]

You're wrong! In our case, all instances are alive and in the same state when both functions are called. It's just that the function get_candidates does not make a hard enough check. So, stateful failover and rpc module calls will not work

[filonenko-mikhail]

get_candidates does not make a hard enough check

What kind of checks do you mean?

[hackallcode]

What kind of checks do you mean?

I don't remember a place with this check, but if you make same actions as in issue, you will get error message, generated by this check