TestTSOKeyspaceGroupSplit is unstable

[lhy1024]

Flaky Test

Which jobs are failing

2023-09-04T07:26:19.6017859Z --- FAIL: TestTSOKeyspaceGroupManager (71.12s)
2023-09-04T07:26:19.6018387Z     --- PASS: TestTSOKeyspaceGroupManager/TestKeyspaceGroupMergeIntoDefault (22.68s)
2023-09-04T07:26:19.6018910Z     --- PASS: TestTSOKeyspaceGroupManager/TestKeyspacesServedByDefaultKeyspaceGroup (3.24s)
2023-09-04T07:26:19.6019416Z     --- PASS: TestTSOKeyspaceGroupManager/TestKeyspacesServedByNonDefaultKeyspaceGroups (3.28s)
2023-09-04T07:26:19.6019860Z     --- PASS: TestTSOKeyspaceGroupManager/TestTSOKeyspaceGroupMembers (0.23s)
2023-09-04T07:26:19.6020250Z     --- PASS: TestTSOKeyspaceGroupManager/TestTSOKeyspaceGroupMerge (5.34s)
2023-09-04T07:26:19.6020706Z     --- PASS: TestTSOKeyspaceGroupManager/TestTSOKeyspaceGroupMergeBeforeInitTSO (6.26s)
2023-09-04T07:26:19.6021133Z     --- PASS: TestTSOKeyspaceGroupManager/TestTSOKeyspaceGroupMergeClient (6.25s)
2023-09-04T07:26:19.6021528Z     --- FAIL: TestTSOKeyspaceGroupManager/TestTSOKeyspaceGroupSplit (20.25s)
2023-09-04T07:26:19.6021945Z     --- PASS: TestTSOKeyspaceGroupManager/TestTSOKeyspaceGroupSplitClient (1.48s)
2023-09-04T07:26:19.6022385Z     --- PASS: TestTSOKeyspaceGroupManager/TestTSOKeyspaceGroupSplitElection (0.77s)

2023-09-04T07:26:19.5511247Z     testutil.go:172: 
2023-09-04T07:26:19.5512213Z         	Error Trace:	/home/runner/work/pd/pd/tests/server/apiv2/handlers/testutil.go:172
2023-09-04T07:26:19.5513284Z         	            				/home/runner/work/pd/pd/tests/integrations/mcs/tso/keyspace_group_manager_test.go:274
2023-09-04T07:26:19.5514063Z         	            				/opt/hostedtoolcache/go/1.21.0/x64/src/runtime/asm_amd64.s:1650
2023-09-04T07:26:19.5514284Z         	Error:      	Not equal: 
2023-09-04T07:26:19.5514578Z         	            	expected: 200
2023-09-04T07:26:19.5515221Z         	            	actual  : 500
2023-09-04T07:26:19.5515599Z         	Test:       	TestTSOKeyspaceGroupManager/TestTSOKeyspaceGroupSplit

2023-09-04T07:26:19.5676578Z     testutil.go:63: 
2023-09-04T07:26:19.5676966Z         	Error Trace:	/home/runner/work/pd/pd/pkg/utils/testutil/testutil.go:63
2023-09-04T07:26:19.5677825Z         	            				/home/runner/work/pd/pd/tests/integrations/mcs/tso/keyspace_group_manager_test.go:273
2023-09-04T07:26:19.5678288Z         	Error:      	Condition never satisfied
2023-09-04T07:26:19.5678654Z         	Test:       	TestTSOKeyspaceGroupManager/TestTSOKeyspaceGroupSplit

CI link

https://github.com/tikv/pd/actions/runs/6070270990/job/16466015972?pr=7036

Reason for failure (if possible)

Anything else

[lhy1024]

A new failure:

2023-11-20T08:03:22.0841565Z panic: Fail in goroutine after TestTSOKeyspaceGroupManager/TestTSOKeyspaceGroupSplit has completed
2023-11-20T08:03:22.0842160Z 
2023-11-20T08:03:22.0842277Z goroutine 1177135 [running]:
2023-11-20T08:03:22.0842584Z testing.(*common).Fail(0xc0039d1ba0)
2023-11-20T08:03:22.0843090Z 	/opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:952 +0x194
2023-11-20T08:03:22.0843747Z testing.(*common).Errorf(0xc0039d1ba0, {0x4487a81, 0x3}, {0xc007669fb0, 0x1, 0x1})
2023-11-20T08:03:22.0844409Z 	/opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1069 +0xa5
2023-11-20T08:03:22.0845197Z github.com/stretchr/testify/assert.Fail({0x523e880, 0xc0039d1ba0}, {0x44c3a6c, 0x19}, {0x0, 0x0, 0x0})
2023-11-20T08:03:22.0846103Z 	/home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.2/assert/assertions.go:262 +0x3ec
2023-11-20T08:03:22.0847061Z github.com/stretchr/testify/assert.Eventually({0x523e880, 0xc0039d1ba0}, 0xc0014cc420, 0x0?, 0x1077b85?, {0x0, 0x0, 0x0})
2023-11-20T08:03:22.0848036Z 	/home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.2/assert/assertions.go:1734 +0x3e5
2023-11-20T08:03:22.0849019Z github.com/stretchr/testify/require.Eventually({0x5246120, 0xc0039d1ba0}, 0xc0039d1ba0?, 0x770fa50?, 0x1?, {0x0, 0x0, 0x0})
2023-11-20T08:03:22.0849979Z 	/home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.2/require/require.go:361 +0xb6
2023-11-20T08:03:22.0851040Z github.com/stretchr/testify/require.(*Assertions).Eventually(0xc006d40e80, 0x20?, 0x7f5587f503b0?, 0x7f558899da68?, {0x0, 0x0, 0x0})
2023-11-20T08:03:22.0852094Z 	/home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.2/require/require_forward.go:289 +0xbf
2023-11-20T08:03:22.0853050Z github.com/tikv/pd/pkg/utils/testutil.Eventually(0x29a1da5?, 0x29a1d8b?, {0xc0029f4d78, 0x2, 0xc0029f4ca8?})
2023-11-20T08:03:22.0853861Z 	/home/runner/work/pd/pd/pkg/utils/testutil/testutil.go:64 +0x18a
2023-11-20T08:03:22.0854604Z github.com/tikv/pd/tests.(*TestTSOCluster).WaitForPrimaryServing(0xc003ff7bc0, 0x0?, 0xde, 0x1)
2023-11-20T08:03:22.0855298Z 	/home/runner/work/pd/pd/tests/tso_cluster.go:168 +0x2b2
2023-11-20T08:03:22.0856176Z github.com/tikv/pd/tests/integrations/mcs/tso.(*tsoKeyspaceGroupManagerTestSuite).requestTSO(0xc00040cee0, 0x0?, 0x9dfa480?, 0xc0?)
2023-11-20T08:03:22.0857285Z 	/home/runner/work/pd/pd/tests/integrations/mcs/tso/keyspace_group_manager_test.go:289 +0xa6
2023-11-20T08:03:22.0858340Z github.com/tikv/pd/tests/integrations/mcs/tso.(*tsoKeyspaceGroupManagerTestSuite).TestTSOKeyspaceGroupSplit.func1()
2023-11-20T08:03:22.0859385Z 	/home/runner/work/pd/pd/tests/integrations/mcs/tso/keyspace_group_manager_test.go:260 +0x65
2023-11-20T08:03:22.0860077Z github.com/stretchr/testify/assert.Eventually.func1()
2023-11-20T08:03:22.0860770Z 	/home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.2/assert/assertions.go:1737 +0x34
2023-11-20T08:03:22.0861542Z created by github.com/stretchr/testify/assert.Eventually in goroutine 1176205
2023-11-20T08:03:22.0862465Z 	/home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.2/assert/assertions.go:1737 +0x376
2023-11-20T08:03:22.0863156Z FAIL	github.com/tikv/pd/tests/integrations/mcs/tso	126.968s
2023-11-20T08:03:22.0863562Z FAIL

https://github.com/tikv/pd/actions/runs/6927028596/job/18840313097