Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Quarkus main] integration test 'main' fails in native mode: QuarkusTestCallbacksITCase.testCallbackContextIsNotFailed #37809

Closed
jerboaa opened this issue Dec 18, 2023 · 6 comments · Fixed by #38127
Assignees
Labels
area/housekeeping Issue type for generalized tasks not related to bugs or enhancements area/native-image area/testing kind/bug Something isn't working
Milestone

Comments

@jerboaa
Copy link
Contributor

jerboaa commented Dec 18, 2023

Describe the bug

We see this failure with various Mandrel versions. 23.1, 22.3 and Graal master:

Error:  Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.015 s <<< FAILURE! -- in io.quarkus.it.main.QuarkusTestCallbacksITCase
Error:  io.quarkus.it.main.QuarkusTestCallbacksITCase.testCallbackContextIsNotFailed -- Time elapsed: 0.003 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <false> but was: <true>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertFalse.failNotFalse(AssertFalse.java:63)
	at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:36)
	at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:31)
	at org.junit.jupiter.api.Assertions.assertFalse(Assertions.java:231)
	at io.quarkus.it.main.QuarkusTestCallbacksITCase.testCallbackContextIsNotFailed(QuarkusTestCallbacksITCase.java:61)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

22.3 failure: https://github.com/graalvm/mandrel/actions/runs/7247202176/job/19741555739?pr=638#step:12:651
23.1 failure: https://github.com/graalvm/mandrel/actions/runs/7228893522/job/19699470818#step:12:706
Graal master failure: https://github.com/graalvm/mandrel/actions/runs/7242468774/job/19728465877#step:12:688

@jerboaa jerboaa added the kind/bug Something isn't working label Dec 18, 2023
Copy link

quarkus-bot bot commented Dec 18, 2023

/cc @Karm (mandrel), @galderz (mandrel), @zakkak (mandrel,native-image)

@jerboaa
Copy link
Contributor Author

jerboaa commented Jan 8, 2024

Is anybody looking at this? We see considerable CI noise due to it. Why is that flagged housekeeping and area/testing? Is it a test issue?

@zakkak
Copy link
Contributor

zakkak commented Jan 9, 2024

I will have a look and try to understand why it's not happening with the published mandrel 23.1 builder image but still shows up in our CI when testing 23.1 source builds.

@zakkak
Copy link
Contributor

zakkak commented Jan 9, 2024

The failures at hand are related to either:

a) a failure in another test, e.g. in io.quarkus.it.main.ImageMetricsITCase#verifyImageMetrics
b) a test abort due to a failing assumption, e.g. in

This happens because test failures are recorded in the test's extension context through io.quarkus.test.junit.AbstractQuarkusTestWithContextExtension and are then seen by io.quarkus.it.main.QuarkusTestCallbacksITCase#testCallbackContextIsNotFailed which checks to see the test status of the context in

assertFalse(TestContextCheckerBeforeEachCallback.CONTEXT.getTestStatus().isTestFailed());

@zakkak
Copy link
Contributor

zakkak commented Jan 9, 2024

@geoand why is Quarkus keeping failures from previous tests in the extension context? I guess the callback test is only interested in failures caused in its test class, no?

Furthermore, isn't this way holding only the latest exception? Wouldn't we be interested in all exceptions instead in the case of a failure?

Note: I am considering ignoring failures caused by TestAbortedException in io.quarkus.it.main.QuarkusTestCallbacksITCase#testCallbackContextIsNotFailed (draft PR #38115) but I would prefer resetting the "state" for each IT. WDYT?

@zakkak zakkak closed this as completed in 789b52f Jan 9, 2024
@zakkak zakkak reopened this Jan 10, 2024
@geoand
Copy link
Contributor

geoand commented Jan 10, 2024

I guess the callback test is only interested in failures caused in its test class, no?

exactly

zakkak added a commit to zakkak/quarkus that referenced this issue Jan 10, 2024
Fix Quarkus TestExtensions to avoid mistakenly transferring the failed
state of one test class to the other.

Closes quarkusio#37809
zakkak added a commit to zakkak/quarkus that referenced this issue Jan 10, 2024
Fix Quarkus TestExtensions to avoid mistakenly transferring the failed
state of one test class to the other.

Closes quarkusio#37809
zakkak added a commit to zakkak/quarkus that referenced this issue Jan 11, 2024
Fix Quarkus TestExtensions to avoid mistakenly transferring the failed
state of one test class to the other.

Closes quarkusio#37809
zakkak added a commit to zakkak/quarkus that referenced this issue Jan 11, 2024
Fix Quarkus TestExtensions to avoid mistakenly transferring the failed
state of one test class to the other.

Closes quarkusio#37809
zakkak added a commit to zakkak/quarkus that referenced this issue Jan 12, 2024
Fix Quarkus TestExtensions to avoid mistakenly transferring the failed
state of one test class to the other.

Closes quarkusio#37809
zakkak added a commit to zakkak/quarkus that referenced this issue Jan 12, 2024
Fix Quarkus TestExtensions to avoid mistakenly transferring the failed
state of one test class to the other.

Closes quarkusio#37809
zakkak added a commit to zakkak/quarkus that referenced this issue Jan 15, 2024
Fix Quarkus TestExtensions to avoid mistakenly transferring the failed
state of one test class to the other.

Closes quarkusio#37809
zakkak added a commit to zakkak/quarkus that referenced this issue Jan 16, 2024
Fix Quarkus TestExtensions to avoid mistakenly transferring the failed
state of one test class to the other.

Closes quarkusio#37809
zakkak added a commit to zakkak/quarkus that referenced this issue Jan 16, 2024
Fix Quarkus TestExtensions to avoid mistakenly transferring the failed
state of one test class to the other.

Closes quarkusio#37809
zakkak added a commit to zakkak/quarkus that referenced this issue Jan 17, 2024
Fix Quarkus TestExtensions to avoid mistakenly transferring the failed
state of one test class to the other.

Closes quarkusio#37809
zakkak added a commit to zakkak/quarkus that referenced this issue Jan 17, 2024
Fix Quarkus TestExtensions to avoid mistakenly transferring the failed
state of one test class to the other.

Closes quarkusio#37809
zakkak added a commit to zakkak/quarkus that referenced this issue Jan 18, 2024
Fix Quarkus TestExtensions to avoid mistakenly transferring the failed
state of one test class to the other.

Closes quarkusio#37809
zakkak added a commit to zakkak/quarkus that referenced this issue Jan 18, 2024
Fix Quarkus TestExtensions to avoid mistakenly transferring the failed
state of one test class to the other.

Closes quarkusio#37809
@quarkus-bot quarkus-bot bot added this to the 3.9 - main milestone Jan 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/housekeeping Issue type for generalized tasks not related to bugs or enhancements area/native-image area/testing kind/bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants