[BUG] Flaky org.opensearch.threadpool.ScalingThreadPoolTests.testScalingThreadPoolConfiguration test

[reta]

Describe the bug
The org.opensearch.threadpool.ScalingThreadPoolTests.testScalingThreadPoolConfiguration test became flaky after #7400

To Reproduce

./gradlew ':server:test' --tests "org.opensearch.threadpool.ScalingThreadPoolTests.testScalingThreadPoolConfiguration" -Dtests.seed=6EC695D9A055C669 

If the commit (#7400) is reverted, test passes normally (seems to be caused by REMOTE_REFRESH thread pool)

Expected behavior
Test should always pass

Plugins
Standard

Screenshots

• https://build.ci.opensearch.org/job/gradle-check/15255
• https://build.ci.opensearch.org/job/gradle-check/15143/

Host/Environment (please complete the following information):

• CI

Additional context
N/A

[reta]

@ashking94 could you please take a look?

[ashking94]

@ashking94 could you please take a look?

Ack

[ashking94]

thanks for creating this @reta. Here is the fix pr - #7539. Do take a look.

[dblock]

This is back in #10430.

[ashking94]

This is back in #10430.

Happened due to some other reason this time, should we create another issue?

[dblock]

@ashking94 Feel free to open another one and (I can) close this one.

[ashking94]

I was able to reproduce this on the commit id where this failed and it was happening due to Names.REMOTE_RECOVERY not being handled in the expectedSize method. This has been handled subsequently recently with #10750.

OpenSearch/server/src/test/java/org/opensearch/threadpool/ScalingThreadPoolTests.java

Lines 143 to 158 in 8673fa9

	private int expectedSize(final String threadPoolName, final int numberOfProcessors) {
	final Map<String, Function<Integer, Integer>> sizes = new HashMap<>();
	sizes.put(ThreadPool.Names.GENERIC, n -> ThreadPool.boundedBy(4 * n, 128, 512));
	sizes.put(ThreadPool.Names.MANAGEMENT, n -> 5);
	sizes.put(ThreadPool.Names.FLUSH, ThreadPool::halfAllocatedProcessorsMaxFive);
	sizes.put(ThreadPool.Names.REFRESH, ThreadPool::halfAllocatedProcessorsMaxTen);
	sizes.put(ThreadPool.Names.WARMER, ThreadPool::halfAllocatedProcessorsMaxFive);
	sizes.put(ThreadPool.Names.SNAPSHOT, ThreadPool::halfAllocatedProcessorsMaxFive);
	sizes.put(ThreadPool.Names.FETCH_SHARD_STARTED, ThreadPool::twiceAllocatedProcessors);
	sizes.put(ThreadPool.Names.FETCH_SHARD_STORE, ThreadPool::twiceAllocatedProcessors);
	sizes.put(ThreadPool.Names.TRANSLOG_TRANSFER, ThreadPool::halfAllocatedProcessorsMaxTen);
	sizes.put(ThreadPool.Names.TRANSLOG_SYNC, n -> 4 * n);
	sizes.put(ThreadPool.Names.REMOTE_PURGE, ThreadPool::halfAllocatedProcessorsMaxFive);
	sizes.put(ThreadPool.Names.REMOTE_REFRESH_RETRY, ThreadPool::halfAllocatedProcessorsMaxTen);
	sizes.put(ThreadPool.Names.REMOTE_RECOVERY, ThreadPool::twiceAllocatedProcessors);
	return sizes.get(threadPoolName).apply(numberOfProcessors);

Also, this has not shown up from multiple days.