[BUG] org.opensearch.search.SearchTimeoutIT.testSimpleTimeout fails intermittently

[jed326]

References
#9378 (comment)
#7357 (comment)
https://build.ci.opensearch.org/job/gradle-check/22854/testReport/junit/org.opensearch.search/SearchTimeoutIT/testSimpleTimeout__p0___search_concurrent_segment_search_enabled___true___/

To Reproduce
Not able to reproduce locally, however this is the reproduction seed provided by the jenkins job.

./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.SearchTimeoutIT" -Dtests.method="testSimpleTimeout {p0={"search.concurrent_segment_search.enabled":"true"}}" -Dtests.seed=4A4EAA35C9A6ED23 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=id -Dtests.timezone=Africa/Algiers -Druntime.java=20

Additional Details
The assertion that is failing is asserting that there were no shard failures in the search request. This assertion failing means that for whatever reason one a shard encountered an exception executing the search request.

[jed326]

From: https://build.ci.opensearch.org/job/gradle-check/28492

This issue does not seem specific to concurrent search

[andrross]

Another failure here:

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.SearchTimeoutIT" -Dtests.method="testSimpleTimeout {p0={"search.concurrent_segment_search.enabled":"true"}}" -Dtests.seed=23BDB16F09456619 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ca -Dtests.timezone=America/Boise -Druntime.java=21

SearchTimeoutIT > testSimpleTimeout {p0={"search.concurrent_segment_search.enabled":"true"}} FAILED
    java.lang.AssertionError: expected:<0> but was:<1>
        at __randomizedtesting.SeedInfo.seed([23BDB16F09456619:60E758393F35311A]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.opensearch.search.SearchTimeoutIT.testSimpleTimeout(SearchTimeoutIT.java:98)

[rajiv-kv]

Failure here

[jainankitk]

Another run: https://build.ci.opensearch.org/job/gradle-check/48025/testReport/junit/org.opensearch.search/SearchTimeoutIT/testSimpleTimeout__p0___search_concurrent_segment_search_enabled___true___/