[BUG] org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringFetchPhase is flaky #7980

BhumikaSaini-Amazon · 2023-06-09T04:37:17Z

Describe the bug
org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringFetchPhase is flaky

https://build.ci.opensearch.org/job/gradle-check/17118/

#7953 (comment)

REPRODUCE WITH: ./gradlew ':qa:smoke-test-http:integTest' --tests "org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringFetchPhase" -Dtests.seed=5306B9DC1B9C7E73 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=de-DE -Dtests.timezone=IET -Druntime.java=20

java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([5306B9DC1B9C7E73:5048BB3B8D45DC09]:0)
	at org.junit.Assert.fail(Assert.java:87)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.junit.Assert.assertTrue(Assert.java:53)
	at org.opensearch.http.SearchRestCancellationIT.lambda$ensureSearchTaskIsCancelled$0(SearchRestCancellationIT.java:220)
	at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1084)
	at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1057)
	at org.opensearch.http.SearchRestCancellationIT.ensureSearchTaskIsCancelled(SearchRestCancellationIT.java:216)
	at org.opensearch.http.SearchRestCancellationIT.verifyCancellationDuringFetchPhase(SearchRestCancellationIT.java:188)
	at org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringFetchPhase(SearchRestCancellationIT.java:162)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:1623)
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.http.SearchRestCancellationIT.lambda$ensureSearchTaskIsCancelled$0(SearchRestCancellationIT.java:220)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 42 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.http.SearchRestCancellationIT.lambda$ensureSearchTaskIsCancelled$0(SearchRestCancellationIT.java:220)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 42 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.http.SearchRestCancellationIT.lambda$ensureSearchTaskIsCancelled$0(SearchRestCancellationIT.java:220)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 42 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.http.SearchRestCancellationIT.lambda$ensureSearchTaskIsCancelled$0(SearchRestCancellationIT.java:220)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 42 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.http.SearchRestCancellationIT.lambda$ensureSearchTaskIsCancelled$0(SearchRestCancellationIT.java:220)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 42 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.http.SearchRestCancellationIT.lambda$ensureSearchTaskIsCancelled$0(SearchRestCancellationIT.java:220)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 42 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.http.SearchRestCancellationIT.lambda$ensureSearchTaskIsCancelled$0(SearchRestCancellationIT.java:220)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 42 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.http.SearchRestCancellationIT.lambda$ensureSearchTaskIsCancelled$0(SearchRestCancellationIT.java:220)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 42 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.http.SearchRestCancellationIT.lambda$ensureSearchTaskIsCancelled$0(SearchRestCancellationIT.java:220)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 42 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.http.SearchRestCancellationIT.lambda$ensureSearchTaskIsCancelled$0(SearchRestCancellationIT.java:220)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 42 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.http.SearchRestCancellationIT.lambda$ensureSearchTaskIsCancelled$0(SearchRestCancellationIT.java:220)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 42 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.http.SearchRestCancellationIT.lambda$ensureSearchTaskIsCancelled$0(SearchRestCancellationIT.java:220)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 42 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.http.SearchRestCancellationIT.lambda$ensureSearchTaskIsCancelled$0(SearchRestCancellationIT.java:220)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 42 more

The text was updated successfully, but these errors were encountered:

stephen-crawford · 2023-06-09T14:32:11Z

I am looking at this issue now but having trouble reproducing it locally.

stephen-crawford · 2023-06-09T21:32:46Z

#7985 should fix this.

reta · 2024-01-05T22:59:13Z

The issue is not fixed (or is back): https://build.ci.opensearch.org/job/gradle-check/31797/testReport/junit/org.opensearch.http/SearchRestCancellationIT/testAutomaticCancellationMultiSearchDuringFetchPhase/

reta · 2024-06-19T15:54:07Z

Closing in favour of #14311

BhumikaSaini-Amazon added bug Something isn't working untriaged labels Jun 9, 2023

BhumikaSaini-Amazon mentioned this issue Jun 9, 2023

Refactor Translog metadata upload/download to write/read header and footer via VersionedCodecStreamWrapper #7953

Merged

6 tasks

stephen-crawford added the flaky-test Random test failure that succeeds on second run label Jun 9, 2023

stephen-crawford mentioned this issue Jun 9, 2023

Fix flaky test SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringFetchPhase #7985

Merged

6 tasks

stephen-crawford removed the untriaged label Jun 12, 2023

dblock closed this as completed in #7985 Jun 13, 2023

reta reopened this Jan 5, 2024

github-actions bot added the untriaged label Jan 5, 2024

reta mentioned this issue Jan 5, 2024

[AUTOCUT] Gradle Check Failure on push to main #11783

Closed

anasalkouz added the Other label Jan 11, 2024

fahadshamiinsta mentioned this issue Jan 12, 2024

Support for Google Application Default Credentials #8394

Merged

6 tasks

vikasvb90 added Search Search query, autocomplete ...etc Other and removed Other labels Jan 30, 2024

msfroh removed the untriaged label Feb 7, 2024

andrross removed the Other label Feb 21, 2024

reta closed this as completed Jun 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringFetchPhase is flaky #7980

[BUG] org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringFetchPhase is flaky #7980

BhumikaSaini-Amazon commented Jun 9, 2023

stephen-crawford commented Jun 9, 2023

stephen-crawford commented Jun 9, 2023

reta commented Jan 5, 2024

reta commented Jun 19, 2024

[BUG] org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringFetchPhase is flaky #7980

[BUG] org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringFetchPhase is flaky #7980

Comments

BhumikaSaini-Amazon commented Jun 9, 2023

stephen-crawford commented Jun 9, 2023

stephen-crawford commented Jun 9, 2023

reta commented Jan 5, 2024

reta commented Jun 19, 2024