failover_promote returns an error via force_inconsistency: true

[dokshina]

Bug description

failover_promote returns an error via force_inconsistency: true. Reproduced in both stateboard cases (tarantool and etcd).
Vclockkeeper value in stateboard storage was changed between get_vclockkeeper and set_vclockkeeper calls in the force_inconsistency function.
That check should be on the server (stateboard or etcd), not the client.

cartridge/cartridge/stateboard-client.lua

Lines 113 to 118 in 5142624

	if keeper.instance_uuid == instance_uuid
	and vclock == nil
	then
	-- No update needed
	return true
	end

Needs extra investigation to fix etcd error.

Steps to reproduce

1. The cluster with simple topology is deployed:

    core_1_replicaset:
      hosts:
        core-1:
      vars:
        replicaset_alias: core-1
        roles:
          - vshard-router
          - failover-coordinator

    storage_1_replicaset:
      hosts:
        storage-1-leader:
        storage-1-replica:
        storage-1-replica-2:
      vars:
        replicaset_alias: storage-1
        failover_priority:
          - storage-1-leader
          - storage-1-replica-2
          - storage-1-replica
        roles:
          - vshard-storage

    storage_2_replicaset:
      hosts:
        storage-2-leader:
        storage-2-replica:
      vars:
        replicaset_alias: storage-2
        failover_priority:
          - storage-2-leader
          - storage-2-replica
        roles:
          - vshard-storage

The failover is set to

    cartridge_failover_params:
      mode: stateful
      state_provider: stateboard
      stateboard_params:
        uri: vm1:4001
        password: secret-stateboard

2. Leaders are promoted to storage-1-replica and storage-2-replica with force_inconsistency: false.
3. Leaders are promoted to core-1, storage-1, storage-2 with force_inconsistency: true.

On step 3 the error is returned: Failed to promote leaders: Promotion succeeded, but inconsistency wasn't forced: Ordinal comparison failed (requested 5, current 7).

1. use test from Add stateful test for promote from ticket 1399 #1710
2. add fiber.sleep before

   cartridge/cartridge/stateboard-client.lua

   Line 122 in 869a2bd

   'set_vclockkeeper', {

3. Profit

Actual behavior

Promotion succeeded, but inconsistency wasn't forced: Compare failed (101): [223 != 253]

cartridge/cartridge/etcd2-client.lua

Line 219 in 869a2bd

local resp, err = session.connection:request('PUT',

Promotion succeeded, but inconsistency wasn't forced: Ordinal comparison failed (requested 5, current 7)

cartridge/cartridge/stateboard-client.lua

Line 122 in 869a2bd

'set_vclockkeeper', {

Expected behavior

No error returned.

[rosik]

Related to #1398

[filonenko-mikhail]

Already covered by tests in other pr #1682

[opomuc]

The issue is still reproduced. @filonenko-mikhail DM me for details, please

[filonenko-mikhail]

Please provide reproducer:

• cartridge version
• env
• step to reproduce

[yngvar-antonsson]

Promotion succeeded, but inconsistency wasn't forced: Compare failed (101): [223 != 253]
It seems that the value by last index (prevIndex) was changed between get_vclockkeeper and set_vclockkeeper in

cartridge/cartridge/etcd2-client.lua

Line 219 in 869a2bd

local resp, err = session.connection:request('PUT',

the same in

cartridge/cartridge/stateboard-client.lua

Line 122 in 869a2bd

'set_vclockkeeper', {

I'll try to write a repro test

[yngvar-antonsson]

Great thanks to @rosik for help to investigate the problem!