kvserver: fail gracefully in TestLeaseTransferRejectedIfTargetNeedsSnapshot

[tbg]

We saw this test hang in CI. What likely happened (according to the stacks) is
that a lease transfer that was supposed to be caught by an interceptor never
showed up in the interceptor. The most likely explanation is that it errored
out before it got to evaluation. It then signaled a channel the test was only
prepared to check later, so the test hung (waiting for a channel that was now
never to be touched).

This test is hard to maintain. It would be great (though, for now, out of reach)
to write tests like it in a deterministic framework¹

For now, fix the test so that when the (so far unknown) error rears its
head again, it will fail properly, so we get to see the error and can
take another pass at fixing the test (separately). Stressing
this commit², we get:

transferErrC unexpectedly signaled: /Table/Max: transfer lease unexpected
error: refusing to transfer lease to (n3,s3):3 because target may need a Raft
snapshot: replica in StateProbe

This makes sense. The test wants to exercise the below-raft mechanism, but
the above-raft mechanism also exists and while we didn't want to interact
with it, we sometimes do¹

The second commit introduces a testing knob that disables the above-raft
mechanism selectively. I've stressed the test for 15 minutes without issues
after this change.

Fixes #106383.

Epic: None
Release note: None

Footnotes

1. see https://github.com/cockroachdb/cockroach/issues/105177. ↩ ↩²

2. ./dev test --filter TestLeaseTransferRejectedIfTargetNeedsSnapshot --stress ./pkg/kv/kvserver/ on gceworker, 285s ↩

[cockroach-teamcity]

This change is 

[erikgrinaker]

The new testing knob isn't necessary, you can use AdminTransferLeaseBypassingSafetyChecks:

cockroach/pkg/kv/db.go

Lines 666 to 675 in adfa97f

	// AdminTransferLeaseBypassingSafetyChecks is like AdminTransferLease, but
	// configures the lease transfer to bypass safety checks. See the comment on
	// AdminTransferLeaseRequest.BypassSafetyChecks for details.
	func (db *DB) AdminTransferLeaseBypassingSafetyChecks(
	ctx context.Context, key interface{}, target roachpb.StoreID,
	) error {
	b := &Batch{}
	b.adminTransferLease(key, target, true /* bypassSafetyChecks */)
	return getOneErr(db.Run(ctx, b), b)
	}

[tbg]

The new testing knob isn't necessary, you can use AdminTransferLeaseBypassingSafetyChecks:

That disables both checks (above and below raft), but I need the below-raft to stay active.

TFTR!

bors r=erikgrinaker

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)