roachtest: loqrecovery/workload=movr/rangeSize=2mb failed

[cockroach-teamcity]

roachtest.loqrecovery/workload=movr/rangeSize=2mb failed with artifacts on master @ 5c2c62ecf1bea60c807edc6b4da22d900ad4ae03:

test artifacts and logs in: /artifacts/loqrecovery/workload=movr/rangeSize=2mb/run_1
	assertions.go:264,assertions.go:1392,require.go:1261,loss_of_quorum_recovery.go:296,monitor.go:105,errgroup.go:74: 
			Error Trace:	/go/src/github.com/cockroachdb/cockroach/loss_of_quorum_recovery.go:296
			            				/go/src/github.com/cockroachdb/cockroach/monitor.go:105
			            				/go/src/github.com/cockroachdb/cockroach/errgroup.go:74
			            				/go/src/github.com/cockroachdb/cockroach/asm_amd64.s:1571
			Error:      	Received unexpected error:
			            	dial tcp 34.75.34.5:26257: connect: connection refused
			Test:       	loqrecovery/workload=movr/rangeSize=2mb

	test_impl.go:265,require.go:1264,loss_of_quorum_recovery.go:296,monitor.go:105,errgroup.go:74: 

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-18725

[irfansharif]

Error appears here:

cockroach/pkg/cmd/roachtest/tests/loss_of_quorum_recovery.go

Line 296 in 3f60a83

require.NoError(t, setSnapshotRate(db, 512))

[cockroach-teamcity]

roachtest.loqrecovery/workload=movr/rangeSize=2mb failed with artifacts on master @ cb55144cdec54d2a70f074ad64b4eca5e6c6891a:

test artifacts and logs in: /artifacts/loqrecovery/workload=movr/rangeSize=2mb/run_1
	assertions.go:264,assertions.go:1392,require.go:1261,loss_of_quorum_recovery.go:296,monitor.go:105,errgroup.go:74: 
			Error Trace:	/go/src/github.com/cockroachdb/cockroach/loss_of_quorum_recovery.go:296
			            				/go/src/github.com/cockroachdb/cockroach/monitor.go:105
			            				/go/src/github.com/cockroachdb/cockroach/errgroup.go:74
			            				/go/src/github.com/cockroachdb/cockroach/asm_amd64.s:1571
			Error:      	Received unexpected error:
			            	dial tcp 35.237.154.125:26257: connect: connection refused
			Test:       	loqrecovery/workload=movr/rangeSize=2mb

	test_impl.go:265,require.go:1264,loss_of_quorum_recovery.go:296,monitor.go:105,errgroup.go:74: 

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[aliher1911]

Looks like #85844 PR broke the tests. But I think tests just surfaced a problem there, we don't seem to handle or enum values for actions and it blows up. The actual failure is:

`* ERROR: a panic has occurred!

• panic: unsupported AllocatorAction: remove dead voter
• (1) attached stack trace
• -- stack trace:
• | runtime.gopanic
• | GOROOT/src/runtime/panic.go:838
• | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*ReplicateQueueMetrics).trackResultByAllocatorAction
• | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replicate_queue.go:498
• | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).processOneChange
• | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replicate_queue.go:910
• | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).process
• | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replicate_queue.go:664
• | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica.func1
• | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:980
• | github.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout
• | github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:91
• | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica
• | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:939
• | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2.1
• | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:858
• | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
• | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:489
• | runtime.goexit
• | GOROOT/src/runtime/asm_amd64.s:1571
• Wraps: (2) panic: unsupported AllocatorAction: remove dead voter
• Error types: (1) *withstack.withStack (2) *errutil.leafError
  `

@AlexTalks what do you think? Do we need to ignore unknown actions or don't pass unknown actions to this method? Maybe we need "Other" for missed cases?

[erikgrinaker]

Also seen in #86107.