Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JDK19 java/lang/Thread/virtual/stress/GetStackTraceALot.java#id0 - Segmentation error vmState=0x0002000f #16351

Closed
JasonFengJ9 opened this issue Nov 22, 2022 · 16 comments · Fixed by #16414
Closed

JDK19 java/lang/Thread/virtual/stress/GetStackTraceALot.java#id0 - Segmentation error vmState=0x0002000f #16351

JasonFengJ9 opened this issue Nov 22, 2022 · 16 comments · Fixed by #16414
Labels
comp:vm project:loom Used to track Project Loom related work test failure triageRequired
Milestone
Java 19 0.37

Comments

@JasonFengJ9
Copy link
Member

JasonFengJ9 commented Nov 22, 2022 •
edited
Loading

Failure link

From an internal build(rhel7lert-11):

openjdk version "19.0.1" 2022-10-18
IBM Semeru Runtime Open Edition 19.0.1+10 (build 19.0.1+10)
Eclipse OpenJ9 VM 19.0.1+10 (build master-5e4baa709, JRE 19 Linux ppc64le-64-Bit Compressed References 20221018_78 (JIT enabled, AOT enabled)
OpenJ9   - 5e4baa709
OMR      - fe4c3b9b5
JCL      - 720d535776 based on jdk-19.0.1+10)

Rerun in Grinder - Change TARGET to run only the failed test targets.

Optional info

Failure output (captured from console output)

[2022-11-19T17:54:36.618Z] variation: -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage Mode150
[2022-11-19T17:54:36.618Z] JVM_OPTIONS:  -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -XX:+UseCompressedOops 

[2022-11-19T18:37:57.679Z] TEST: java/lang/Thread/virtual/stress/GetStackTraceALot.java#id0

[2022-11-19T18:37:57.680Z] STDERR:
[2022-11-19T18:37:57.680Z] JVMDUMP034I User requested System dump using '/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_testList_1/aqa-tests/TKG/output_16688804752890/jdk_lang_0/work/scratch/1/core.20221119.103746.5578.0001.dmp' through com.ibm.jvm.Dump.systemDumpToFile
[2022-11-19T18:37:57.680Z] JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_testList_1/aqa-tests/TKG/output_16688804752890/jdk_lang_0/work/scratch/1/core.20221119.103746.5578.0001.dmp
[2022-11-19T18:37:57.680Z] JVMDUMP034I User requested Java dump using '/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_testList_1/aqa-tests/TKG/output_16688804752890/jdk_lang_0/work/scratch/1/javacore.20221119.103748.5578.0002.txt' through com.ibm.jvm.Dump.javaDumpToFile
[2022-11-19T18:37:57.680Z] JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_testList_1/aqa-tests/TKG/output_16688804752890/jdk_lang_0/work/scratch/1/javacore.20221119.103748.5578.0002.txt
[2022-11-19T18:37:57.680Z] Unhandled exception
[2022-11-19T18:37:57.680Z] Type=Segmentation error vmState=0x0002000f
[2022-11-19T18:37:57.680Z] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
[2022-11-19T18:37:57.680Z] Handler1=00003FFFA53B0140 Handler2=00003FFFA52D8AA0
[2022-11-19T18:37:57.680Z] R0=0000000000000008 R1=00003FFF7C36BA20 R2=00003FFFA5C07400 R3=00000000E01B3178
[2022-11-19T18:37:57.680Z] R4=0000000100000000 R5=0000000000520042 R6=0000000000000010 R7=0000000000000020
[2022-11-19T18:37:57.680Z] R8=0000000000000030 R9=00003FFF7C36BCB0 R10=0000000000000008 R11=00000000E01FE700
[2022-11-19T18:37:57.680Z] R12=000000000000A400 R13=00003FFF7C376900 R14=00000000FFFB4A78 R15=00000000FFFB4A78
[2022-11-19T18:37:57.680Z] R16=0000000000000000 R17=0000000000000001 R18=00003FFF7C36BCE0 R19=0000000000000000
[2022-11-19T18:37:57.680Z] R20=00003FFFA0090270 R21=00003FFF7C36BCB0 R22=00000006000B7700 R23=00000000E06D31C8
[2022-11-19T18:37:57.680Z] R24=0000000000000009 R25=0000000000000000 R26=00000000E01B3178 R27=00000000E01B3178
[2022-11-19T18:37:57.680Z] R28=00003FFF7C36BCB0 R29=00003FFF985410D0 R30=00003FFFA008FFC0 R31=00003FFF28001AC8
[2022-11-19T18:37:57.680Z] NIP=00003FFFA5ACABC0 MSR=800000010280F033 ORIG_GPR3=C0000000000027E0 CTR=0000000000009A95
[2022-11-19T18:37:57.680Z] LINK=00003FFFA4DA8C40 XER=0000000000000000 CCR=0000000044844248 SOFTE=0000000000000001
[2022-11-19T18:37:57.680Z] TRAP=0000000000000300 DAR=0000000100000000 dsisr=0000000040000000 RESULT=0000000000000000
[2022-11-19T18:37:57.680Z] FPR0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR1 000000000052057e (f: 5375358.000000, d: 2.655780e-317)
[2022-11-19T18:37:57.680Z] FPR2 3fd6e80000000000 (f: 0.000000, d: 3.579102e-01)
[2022-11-19T18:37:57.680Z] FPR3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR4 3fe3333340000000 (f: 1073741824.000000, d: 6.000000e-01)
[2022-11-19T18:37:57.680Z] FPR5 3fd6e850e0000000 (f: 3758096384.000000, d: 3.579294e-01)
[2022-11-19T18:37:57.680Z] FPR6 3eb742873eb74287 (f: 1052197504.000000, d: 1.386397e-06)
[2022-11-19T18:37:57.680Z] FPR7 40365ba140000000 (f: 1073741824.000000, d: 2.235793e+01)
[2022-11-19T18:37:57.680Z] FPR8 3e66376980000000 (f: 2147483648.000000, d: 4.138137e-08)
[2022-11-19T18:37:57.680Z] FPR9 4000000000000000 (f: 0.000000, d: 2.000000e+00)
[2022-11-19T18:37:57.680Z] FPR10 3fae368d40000000 (f: 1073741824.000000, d: 5.900995e-02)
[2022-11-19T18:37:57.680Z] FPR11 bfa1254480000000 (f: 2147483648.000000, d: -3.348745e-02)
[2022-11-19T18:37:57.680Z] FPR12 4aa40afc4aa40afc (f: 1252264704.000000, d: 3.749472e+51)
[2022-11-19T18:37:57.680Z] FPR13 3fdb9b2820000000 (f: 536870912.000000, d: 4.313450e-01)
[2022-11-19T18:37:57.680Z] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] Module=/lib64/libc.so.6
[2022-11-19T18:37:57.680Z] Module_base_address=00003FFFA5A20000
[2022-11-19T18:37:57.680Z] Target=2_90_20221018_78 (Linux 3.10.0-1160.80.1.el7.ppc64le)
[2022-11-19T18:37:57.680Z] CPU=ppc64le (4 logical CPUs) (0x1de3d0000 RAM)
[2022-11-19T18:37:57.680Z] ----------- Stack Backtrace -----------
[2022-11-19T18:37:57.680Z] protectedBacktrace+0x24 (0x00003FFFA52D2EB4 [libj9prt29.so+0x32eb4])
[2022-11-19T18:37:57.680Z] omrsig_protect+0x3f4 (0x00003FFFA52D9F74 [libj9prt29.so+0x39f74])
[2022-11-19T18:37:57.680Z] omrintrospect_backtrace_thread_raw+0xe0 (0x00003FFFA52D34D0 [libj9prt29.so+0x334d0])
[2022-11-19T18:37:57.680Z] protectedIntrospectBacktraceThread+0x24 (0x00003FFFA52D2AF4 [libj9prt29.so+0x32af4])
[2022-11-19T18:37:57.680Z] omrsig_protect+0x3f4 (0x00003FFFA52D9F74 [libj9prt29.so+0x39f74])
[2022-11-19T18:37:57.680Z] omrintrospect_backtrace_thread+0x94 (0x00003FFFA52D2C04 [libj9prt29.so+0x32c04])
[2022-11-19T18:37:57.680Z] generateDiagnosticFiles+0xd4 (0x00003FFFA53AFB14 [libj9vm29.so+0x3fb14])
[2022-11-19T18:37:57.680Z] omrsig_protect+0x3f4 (0x00003FFFA52D9F74 [libj9prt29.so+0x39f74])
[2022-11-19T18:37:57.680Z] structuredSignalHandler+0x194 (0x00003FFFA53AFE74 [libj9vm29.so+0x3fe74])
[2022-11-19T18:37:57.680Z] structuredSignalHandlerVM+0x58 (0x00003FFFA53B0198 [libj9vm29.so+0x40198])
[2022-11-19T18:37:57.680Z] mainSynchSignalHandler+0x294 (0x00003FFFA52D8D34 [libj9prt29.so+0x38d34])
[2022-11-19T18:37:57.680Z] __kernel_sigtramp_rt64+0x0 (0x00003FFFA5D10478)
[2022-11-19T18:37:57.680Z] _ZN12MM_Scavenger14copyForVariantILb0EEEP8J9ObjectP22MM_EnvironmentStandardP18MM_ForwardedHeader+0x360 (0x00003FFFA4DA8720 [libj9gc29.so+0x1e8720])
[2022-11-19T18:37:57.680Z] _ZN12MM_Scavenger26incrementalScanCacheBySlotEP22MM_EnvironmentStandardP24MM_CopyScanCacheStandard+0x978 (0x00003FFFA4DA4F18 [libj9gc29.so+0x1e4f18])
[2022-11-19T18:37:57.680Z] _ZN12MM_Scavenger12completeScanEP22MM_EnvironmentStandard+0x114 (0x00003FFFA4DA55C4 [libj9gc29.so+0x1e55c4])
[2022-11-19T18:37:57.680Z] _ZN12MM_Scavenger24workThreadGarbageCollectEP22MM_EnvironmentStandard+0x2b4 (0x00003FFFA4DA5AE4 [libj9gc29.so+0x1e5ae4])
[2022-11-19T18:37:57.680Z] _ZN23MM_ParallelScavengeTask3runEP18MM_EnvironmentBase+0x1c (0x00003FFFA4E1DD3C [libj9gc29.so+0x25dd3c])
[2022-11-19T18:37:57.680Z] _ZN21MM_ParallelDispatcher16workerEntryPointEP18MM_EnvironmentBase+0x2b8 (0x00003FFFA4D40328 [libj9gc29.so+0x180328])
[2022-11-19T18:37:57.680Z] _Z23dispatcher_thread_proc2P14OMRPortLibraryPv+0x160 (0x00003FFFA4D3F6F0 [libj9gc29.so+0x17f6f0])
[2022-11-19T18:37:57.680Z] omrsig_protect+0x3f4 (0x00003FFFA52D9F74 [libj9prt29.so+0x39f74])
[2022-11-19T18:37:57.680Z] dispatcher_thread_proc+0x50 (0x00003FFFA4D3EDB0 [libj9gc29.so+0x17edb0])
[2022-11-19T18:37:57.680Z] thread_wrapper+0x190 (0x00003FFFA526CBC0 [libj9thr29.so+0xcbc0])
[2022-11-19T18:37:57.680Z] start_thread+0x104 (0x00003FFFA5C48CD4 [libpthread.so.0+0x8cd4])
[2022-11-19T18:37:57.680Z] clone+0xe4 (0x00003FFFA5B47F14 [libc.so.6+0x127f14])
[2022-11-19T18:37:57.680Z] ---------------------------------------
[2022-11-19T18:37:57.680Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2022/11/19 10:37:49 - please wait.

[2022-11-19T18:37:57.681Z] TEST RESULT: Error. Program `/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_testList_1/openjdkbinary/j2sdk-image/bin/java' timed out (timeout set to 960000ms, elapsed time including timeout handling was 965937ms).
[2022-11-19T18:37:57.681Z] --------------------------------------------------
[2022-11-19T18:43:14.332Z] Test results: passed: 845; error: 1
[2022-11-19T18:43:39.475Z] Report written to /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_testList_1/aqa-tests/TKG/output_16688804752890/jdk_lang_0/report/html/report.html
[2022-11-19T18:43:39.475Z] Results written to /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_testList_1/aqa-tests/TKG/output_16688804752890/jdk_lang_0/work
[2022-11-19T18:43:39.475Z] Error: Some tests failed or other problems occurred.
[2022-11-19T18:43:39.475Z] 
[2022-11-19T18:43:39.475Z] jdk_lang_0_FAILED

50x internal grinder: ubu18le-rt1-6 failure, rhel7lert-1-10 failure

FYI @dmitripivkine
@JasonFengJ9 JasonFengJ9 added this to the Java 19 milestone Nov 22, 2022
@dmitripivkine
Copy link
Contributor

This is another manifestation of crashes in Scavenger we collecting in #16249

> !j9javavm 0x00003FFFA001CEC0 | grep liveVirtualThreadList
	0x4f88: struct J9Object** liveVirtualThreadList = !j9x 0x00003FFFA00A1888
> !j9x 0x00003FFFA00A1888
0x3FFFA00A1888 :  00000000ffd8dbf0 00000000e0109860 [ ........`....... ]

> !j9object 0x00000000ffd8dbf0
!J9Object 0x00000000FFD8DBF0 {
	struct J9Class* clazz = !j9class 0x101C00 // java/lang/VirtualThread
	Object flags = 0x0000002A;
	I lockword = 0x00000008 (offset = 0) (java/lang/Object) <hidden>
	J eetop = 0x0000000000000000 (offset = 4) (java/lang/Thread)
	J tid = 0x0000000000000000 (offset = 12) (java/lang/Thread)
	Ljava/lang/String; name = !fj9object 0x0 (offset = 36) (java/lang/Thread)
	Z interrupted = 0x00000000 (offset = 92) (java/lang/Thread)
	Ljava/lang/ClassLoader; contextClassLoader = !fj9object 0x0 (offset = 40) (java/lang/Thread)
	Ljava/security/AccessControlContext; inheritedAccessControlContext = !fj9object 0x0 (offset = 44) (java/lang/Thread)
	Ljava/lang/Thread$FieldHolder; holder = !fj9object 0x0 (offset = 48) (java/lang/Thread)
	Ljava/lang/ThreadLocal$ThreadLocalMap; threadLocals = !fj9object 0x0 (offset = 52) (java/lang/Thread)
	Ljava/lang/ThreadLocal$ThreadLocalMap; inheritableThreadLocals = !fj9object 0x0 (offset = 56) (java/lang/Thread)
	Ljava/lang/Object; extentLocalBindings = !fj9object 0x0 (offset = 60) (java/lang/Thread)
	Ljava/lang/Object; interruptLock = !fj9object 0x0 (offset = 64) (java/lang/Thread)
	Ljava/lang/Object; parkBlocker = !fj9object 0x0 (offset = 68) (java/lang/Thread)
	Lsun/nio/ch/Interruptible; nioBlocker = !fj9object 0x0 (offset = 72) (java/lang/Thread)
	Ljdk/internal/vm/Continuation; cont = !fj9object 0x0 (offset = 76) (java/lang/Thread)
	Ljava/lang/Thread$UncaughtExceptionHandler; uncaughtExceptionHandler = !fj9object 0x0 (offset = 80) (java/lang/Thread)
	J threadLocalRandomSeed = 0x0000000000000000 (offset = 20) (java/lang/Thread)
	I threadLocalRandomProbe = 0x00000000 (offset = 96) (java/lang/Thread)
	I threadLocalRandomSecondarySeed = 0x00000000 (offset = 100) (java/lang/Thread)
	Ljdk/internal/vm/ThreadContainer; container = !fj9object 0x0 (offset = 84) (java/lang/Thread)
	Ljdk/internal/vm/StackableScope; headStackableScopes = !fj9object 0x0 (offset = 88) (java/lang/Thread)
	Z started = 0x00000000 (offset = 104) (java/lang/Thread)
	Z stopCalled = 0x00000000 (offset = 108) (java/lang/Thread)
	J tls = 0x0000000000000000 (offset = 28) (java/lang/Thread) <hidden>
	Ljava/util/concurrent/Executor; scheduler = !fj9object 0x0 (offset = 124) (java/lang/VirtualThread)
	Ljdk/internal/vm/Continuation; cont = !fj9object 0x0 (offset = 128) (java/lang/VirtualThread)
	Ljava/lang/Runnable; runContinuation = !fj9object 0x0 (offset = 132) (java/lang/VirtualThread)
	I state = 0x00000000 (offset = 112) (java/lang/VirtualThread)
	Z parkPermit = 0x00000000 (offset = 152) (java/lang/VirtualThread)
	Ljava/lang/Thread; carrierThread = !fj9object 0x0 (offset = 136) (java/lang/VirtualThread)
	Ljava/util/concurrent/CountDownLatch; termination = !fj9object 0x0 (offset = 140) (java/lang/VirtualThread)
	Ljava/lang/VirtualThread; linkNext = !fj9object 0xfffb4a78 (offset = 148) (java/lang/VirtualThread) <hidden>
	Ljava/lang/VirtualThread; linkPrevious = !fj9object 0xfffb4a78 (offset = 144) (java/lang/VirtualThread) <hidden>
	J inspectorCount = 0x0000000000000000 (offset = 116) (java/lang/VirtualThread) <hidden>
	I isSuspendedByJVMTI = 0x00000000 (offset = 156) (java/lang/VirtualThread) <hidden>
}
> !fj9object 0xfffb4a78
Unable to read object clazz at 0x00000000FFFB4A78 (clazz = 0x00000000E01B3100)
> !j9x 0xfffb4a00,100,4
0xFFFB4A00 :  fffb1830 fffb1868 fffb18a8 fffb18d8 [ 0...h........... ]
0xFFFB4A10 :  fffb1910 fffb1950 fffb1990 fffb19c0 [ ....P........... ]
0xFFFB4A20 :  fffb1a00 fffb1a38 fffb1c30 fffb1c70 [ ....8...0...p... ]
0xFFFB4A30 :  fffb1ca8 fffb1cd8 fffb1d10 fffb1d50 [ ............P... ]
0xFFFB4A40 :  fffb1d80 fffb1dc0 fffb1df8 fffb1e28 [ ............(... ]
0xFFFB4A50 :  fffb1e60 fffb1ea0 fffb1ed8 fffb1f08 [ `............... ]
0xFFFB4A60 :  fffb1f40 fffb1f78 fffb1fb8 fffb1fe8 [ @...x........... ]
0xFFFB4A70 :  fffb2020 fffb2060 e01b317c 00000000 [   ..` ..|1...... ]
0xFFFB4A80 :  fffb2108 fffb2138 fffb2170 fffb21a8 [ .!..8!..p!...!.. ]
0xFFFB4A90 :  fffb21e8 fffb2220 fffb2260 fffb2290 [ .!.. "..`"...".. ]
0xFFFB4AA0 :  fffb22c8 fffb2300 fffb2348 fffb2380 [ ."...#..H#...#.. ]
0xFFFB4AB0 :  fffb23b8 fffb23f8 fffb26c8 fffb2700 [ .#...#...&...'.. ]
0xFFFB4AC0 :  fffb2740 fffb2778 fffb27a8 fffb27e8 [ @'..x'...'...'.. ]
0xFFFB4AD0 :  fffb2820 fffb2850 fffb2890 fffb28c8 [  (..P(...(...(.. ]
0xFFFB4AE0 :  fffb28f8 fffb2938 fffb2970 fffb29a0 [ .(..8)..p)...).. ]
0xFFFB4AF0 :  fffb29e0 fffb2a18 fffb2a48 fffb2a88 [ .)...*..H*...*.. ]
> !j9object 0xe01b3178
Unable to read object clazz at 0x00000000E01B3178 (clazz = 0x00000000E01B3100)
> !j9x 0x00000000E01B3100,100,4
0xE01B3100 :  00000000 00000000 00000000 00000000 [ ................ ]
0xE01B3110 :  00000000 00000000 00000000 00000000 [ ................ ]
0xE01B3120 :  00000000 00000000 00000000 00000000 [ ................ ]
0xE01B3130 :  00000000 00000000 00000000 00000000 [ ................ ]
0xE01B3140 :  00000000 00000000 00000000 00000000 [ ................ ]
0xE01B3150 :  00000000 00000000 00000000 00000000 [ ................ ]
0xE01B3160 :  00000000 00000000 00000000 00000000 [ ................ ]
0xE01B3170 :  00000000 00000000 e01b317c 00000000 [ ........|1...... ]
0xE01B3180 :  fffb2108 fffb2138 fffb2170 fffb21a8 [ .!..8!..p!...!.. ]
0xE01B3190 :  fffb21e8 fffb2220 fffb2260 fffb2290 [ .!.. "..`"...".. ]
0xE01B31A0 :  fffb22c8 fffb2300 fffb2348 fffb2380 [ ."...#..H#...#.. ]
0xE01B31B0 :  fffb23b8 fffb23f8 fffb26c8 fffb2700 [ .#...#...&...'.. ]
0xE01B31C0 :  fffb2740 fffb2778 fffb27a8 fffb27e8 [ @'..x'...'...'.. ]
0xE01B31D0 :  fffb2820 fffb2850 fffb2890 fffb28c8 [  (..P(...(...(.. ]
0xE01B31E0 :  fffb28f8 fffb2938 fffb2970 fffb29a0 [ .(..8)..p)...).. ]
0xE01B31F0 :  fffb29e0 fffb2a18 fffb2a48 fffb2a88 [ .)...*..H*...*.. ]
>

We are trying to copy bogus object !fj9object 0xfffb4a78 and crashed

Closing as duplicate of #16249

@dmitripivkine dmitripivkine closed this as not planned Won't fix, can't repro, duplicate, stale Nov 22, 2022
@dmitripivkine dmitripivkine mentioned this issue Nov 22, 2022
Closed
@tajila
Copy link
Contributor

tajila commented Nov 22, 2022 •
edited
Loading

10x grinder with trace logs https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/29690/

@tajila
Copy link
Contributor

tajila commented Nov 22, 2022

Ill reopen this as the other issue contains the jit invalid address details. We can keep discussion on corrupt vthread list here

@tajila tajila reopened this Nov 22, 2022
@tajila
Copy link
Contributor

tajila commented Nov 22, 2022

@babsingh can you please take a look

@JasonFengJ9
Copy link
Member Author

The grinder reproduced the Segmentation error vmState=0x0002000f twice: ubu18le-rt1-6 failure, rhel7lert-1-10 failure

Grinder 4(rhel8le-rt1-10) had an assertion j9mm.141 * ** ASSERTION FAILED ** at /home/jenkins/workspace/build-scripts/jobs/jdk19/jdk19-linux-ppc64le-openj9/workspace/build/src/omr/gc/base/standard/Scavenger.cpp:1622: ((false))

09:03:45  STDERR:
09:03:45  JVMDUMP034I User requested System dump using '/home/jenkins/workspace/Grinder_iteration_4/aqa-tests/TKG/output_1669124827859/jdk_custom_0/work/scratch/0/core.20221122.060328.1566414.0001.dmp' through com.ibm.jvm.Dump.systemDumpToFile
09:03:45  JVMDUMP010I System dump written to /home/jenkins/workspace/Grinder_iteration_4/aqa-tests/TKG/output_1669124827859/jdk_custom_0/work/scratch/0/core.20221122.060328.1566414.0001.dmp
09:03:45  JVMDUMP034I User requested Java dump using '/home/jenkins/workspace/Grinder_iteration_4/aqa-tests/TKG/output_1669124827859/jdk_custom_0/work/scratch/0/javacore.20221122.060329.1566414.0002.txt' through com.ibm.jvm.Dump.javaDumpToFile
09:03:45  JVMDUMP010I Java dump written to /home/jenkins/workspace/Grinder_iteration_4/aqa-tests/TKG/output_1669124827859/jdk_custom_0/work/scratch/0/javacore.20221122.060329.1566414.0002.txt
09:03:45  Corruption in Evacuate at 00000000FFFBC1C8: calculated object size 3996785944479924229 larger than available 278072, Forwarded Header at 00007FFF8C3FBC50
09:03:45  14:03:31.006 0x23ca00    j9mm.141    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/build-scripts/jobs/jdk19/jdk19-linux-ppc64le-openj9/workspace/build/src/omr/gc/base/standard/Scavenger.cpp:1622: ((false))
09:03:45  JVMDUMP039I Processing dump event "traceassert", detail "" at 2022/11/22 06:03:31 - please wait.

@tajila
Copy link
Contributor

tajila commented Nov 22, 2022

@JasonFengJ9 can you reproduce with -Xtrace:maximal={j9scar{level5}},buffers=2m, we added some new tracepoints to help diagnose this issue

@tajila tajila added comp:vm project:loom Used to track Project Loom related work labels Nov 22, 2022
@JasonFengJ9
Copy link
Member Author

@tajila 50x grinder -Xtrace:maximal={j9scar{level5}},buffers=2m just was launched.

@dmitripivkine
Copy link
Contributor

Another failure in the grinder (launched for different defect) https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/29676/

> !gpinfo
Failing Thread: !j9vmthread 0x3fef4002400
Failing Thread ID: 0x1db01 (121601)
gpInfo:
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=000003FF94B44858 Handler2=000003FF94A31CF0 InaccessibleAddress=0408040804080000
gpr0=000003FF9027C210 gpr1=000003FF9003E158 gpr2=000003FF9005F728 gpr3=000003FEF4004338
gpr4=000003FF944FE6C0 gpr5=000003FF944FE478 gpr6=000003FF944FE480 gpr7=000003FF900AE170
gpr8=000003FF944FE6D8 gpr9=000003FEF4004338 gpr10=000003FF900AE170 gpr11=0408040804080400
gpr12=000003FF95726F88 gpr13=000003FF944FE6C0 gpr14=000003FF9432C368 gpr15=000003FF944FE2E8
psw=000003FF9435625C mask=0705100180000000 fpc=00080000 bea=000003FF9432C362
fpr0 429841d9d1da893c (f: 3520760064.000000, d: 6.667777e+12)
fpr1 000003ff944fe088 (f: 2488262912.000000, d: 2.172031e-311)
fpr2 3e3ab2831be3a8dd (f: 467904736.000000, d: 6.215953e-09)
fpr3 0000000000110000 (f: 1114112.000000, d: 5.504445e-318)
fpr4 402bb9d3beb8c86b (f: 3199780864.000000, d: 1.386294e+01)
fpr5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr6 3f33240000000000 (f: 0.000000, d: 2.920628e-04)
fpr7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr8 0000000000041000 (f: 266240.000000, d: 1.315400e-318)
fpr9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr10 000003ff944bf000 (f: 2488004608.000000, d: 2.172031e-311)
fpr11 000003ffd3ffb4e0 (f: 3556750592.000000, d: 2.172559e-311)
fpr12 0005ee048c602b82 (f: 2355112960.000000, d: 8.246342e-309)
fpr13 000003fee8045d58 (f: 3892600064.000000, d: 2.170603e-311)
fpr14 000003ff943c65e0 (f: 2486986240.000000, d: 2.172030e-311)
fpr15 000003fee809ab18 (f: 3892947712.000000, d: 2.170603e-311)
Module=/home/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image/lib/default/libj9gc_full29.so
Module_base_address=000003FF94180000

> !findvm
!j9javavm 0x000003FF90038DA0
> !j9javavm 0x000003FF90038DA0 | grep liveVirtualThreadList
	0x4f40: struct J9Object** liveVirtualThreadList = !j9x 0x000003FF900C0388
	0x4f48: struct J9ThreadMonitor* liveVirtualThreadListMutex = !j9threadmonitor 0x000003FF9008E838
> !j9x 0x000003FF900C0388
0x3FF900C0388 :  000003ff6e77c9c0 000003ff6e73a8a8 [ ....nw......ns.. ]
0x3FF900C0398 :  000003ff6e73ac50 000003ff6e73bd70 [ ....ns.P....ns.p ]
> !j9object 0x000003ff6e77c9c0
!J9Object 0x000003FF6E77C9C0 {
	struct J9Class* clazz = !j9class 0x3FF902D7400 // java/lang/VirtualThread
	Object flags = 0x0000000A;
	J lockword = 0x0000000000000000 (offset = 0) (java/lang/Object) <hidden>
	J eetop = 0x0000000000000000 (offset = 8) (java/lang/Thread)
	J tid = 0x0000000000000000 (offset = 16) (java/lang/Thread)
	Ljava/lang/String; name = !fj9object 0x0 (offset = 40) (java/lang/Thread)
	Z interrupted = 0x00000000 (offset = 152) (java/lang/Thread)
	Ljava/lang/ClassLoader; contextClassLoader = !fj9object 0x0 (offset = 48) (java/lang/Thread)
	Ljava/security/AccessControlContext; inheritedAccessControlContext = !fj9object 0x0 (offset = 56) (java/lang/Thread)
	Ljava/lang/Thread$FieldHolder; holder = !fj9object 0x0 (offset = 64) (java/lang/Thread)
	Ljava/lang/ThreadLocal$ThreadLocalMap; threadLocals = !fj9object 0x0 (offset = 72) (java/lang/Thread)
	Ljava/lang/ThreadLocal$ThreadLocalMap; inheritableThreadLocals = !fj9object 0x0 (offset = 80) (java/lang/Thread)
	Ljava/lang/Object; extentLocalBindings = !fj9object 0x0 (offset = 88) (java/lang/Thread)
	Ljava/lang/Object; interruptLock = !fj9object 0x0 (offset = 96) (java/lang/Thread)
	Ljava/lang/Object; parkBlocker = !fj9object 0x0 (offset = 104) (java/lang/Thread)
	Lsun/nio/ch/Interruptible; nioBlocker = !fj9object 0x0 (offset = 112) (java/lang/Thread)
	Ljdk/internal/vm/Continuation; cont = !fj9object 0x0 (offset = 120) (java/lang/Thread)
	Ljava/lang/Thread$UncaughtExceptionHandler; uncaughtExceptionHandler = !fj9object 0x0 (offset = 128) (java/lang/Thread)
	J threadLocalRandomSeed = 0x0000000000000000 (offset = 24) (java/lang/Thread)
	I threadLocalRandomProbe = 0x00000000 (offset = 156) (java/lang/Thread)
	I threadLocalRandomSecondarySeed = 0x00000000 (offset = 160) (java/lang/Thread)
	Ljdk/internal/vm/ThreadContainer; container = !fj9object 0x0 (offset = 136) (java/lang/Thread)
	Ljdk/internal/vm/StackableScope; headStackableScopes = !fj9object 0x0 (offset = 144) (java/lang/Thread)
	Z started = 0x00000000 (offset = 164) (java/lang/Thread)
	Z stopCalled = 0x00000000 (offset = 168) (java/lang/Thread)
	J tls = 0x0000000000000000 (offset = 32) (java/lang/Thread) <hidden>
	Ljava/util/concurrent/Executor; scheduler = !fj9object 0x0 (offset = 184) (java/lang/VirtualThread)
	Ljdk/internal/vm/Continuation; cont = !fj9object 0x0 (offset = 192) (java/lang/VirtualThread)
	Ljava/lang/Runnable; runContinuation = !fj9object 0x0 (offset = 200) (java/lang/VirtualThread)
	I state = 0x00000000 (offset = 172) (java/lang/VirtualThread)
	Z parkPermit = 0x00000000 (offset = 240) (java/lang/VirtualThread)
	Ljava/lang/Thread; carrierThread = !fj9object 0x0 (offset = 208) (java/lang/VirtualThread)
	Ljava/util/concurrent/CountDownLatch; termination = !fj9object 0x0 (offset = 216) (java/lang/VirtualThread)
	Ljava/lang/VirtualThread; linkNext = !fj9object 0x3ff8e3e3d10 (offset = 232) (java/lang/VirtualThread) <hidden>
	Ljava/lang/VirtualThread; linkPrevious = !fj9object 0x3ff8e3e3d10 (offset = 224) (java/lang/VirtualThread) <hidden>
	J inspectorCount = 0x0000000000000000 (offset = 176) (java/lang/VirtualThread) <hidden>
	I isSuspendedByJVMTI = 0x00000000 (offset = 244) (java/lang/VirtualThread) <hidden>
}
> !fj9object 0x3ff8e3e3d10
Unable to read object clazz at 0x000003FF8E3E3D10 (clazz = 0x0408040804080400)
> !j9x 0x000003FF8E3E3c80,100,4
0x3FF8E3E3C80 :  60520324 523ce052 8ba452c5 60530f44 [ `R.$R<.R..R.`S.D ]
0x3FF8E3E3C90 :  53508053 976453db a4541fe4 54642454 [ SP.S.dS..T..Td$T ]
0x3FF8E3E3CA0 :  a86454ec a45530e4 5577c455 bc045600 [ .dT..U0.Uw.U..V. ]
0x3FF8E3E3CB0 :  44564484 5688c456 cd045711 44575584 [ DVD.V..V..W.DWU. ]
0x3FF8E3E3CC0 :  5799c457 de045822 44586684 58ad6458 [ W..W..X"DXf.X.dX ]
0x3FF8E3E3CD0 :  f1a45935 e4597a24 59be645a 02a45a46 [ ..Y5.Yz$Y.dZ..ZF ]
0x3FF8E3E3CE0 :  e45a8b24 5acf645b 13a45b57 e45b9ec4 [ .Z.$Z.d[..[W.[.. ]
0x3FF8E3E3CF0 :  5bed845c 27445c76 047f0000 0d9c0408 [ [..\'D\v........ ]
0x3FF8E3E3D00 :  04080408 04080408 04080408 04080408 [ ................ ]
0x3FF8E3E3D10 :  04080408 04080408 04080408 04080408 [ ................ ] <--- pointer mid-object
0x3FF8E3E3D20 :  04080408 04080408 04080408 04080408 [ ................ ]
0x3FF8E3E3D30 :  04080408 04080408 04080408 04080408 [ ................ ]
0x3FF8E3E3D40 :  04080408 04080408 04080408 04080408 [ ................ ]
0x3FF8E3E3D50 :  04080408 04080402 3e784842 ae784848 [ ........>xHB.xHH ]
0x3FF8E3E3D60 :  000003ff 90286100 0000015c 00000000 [ .....(a....\.... ]
0x3FF8E3E3D70 :  000003ff 8e3e3d78 01000000 07ffffff [ .....>=x........ ]

@babsingh
Copy link
Contributor

babsingh commented Nov 22, 2022 •
edited
Loading

@babsingh can you please take a look

In @JasonFengJ9's latest grinder: https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/29692, all failures are timeouts from #16340. The GC segfault, reported in this issue, is not seen.

Corefiles for @dmitripivkine's grinder: https://na.artifactory.swg-devops.com/artifactory/sys-rt-generic-local/hyc-runtimes-jenkins.swg-devops.com/Grinder/29676/openjdk_test_output.tar.gz

It is not just the liveVirtualThreadList which has CORRUPT VirtualThread objects as mentioned by @dmitripivkine in #16351 (comment).

Even, the Java stack has CORRUPT VirtualThread objects; see the jdmpview output below. The CORRUPT VirtualThread is derived using Thread.currentThread() in Access.parkVirtualThread(...). Thread.currentThread() is JIT inlined; so, the JIT may be involved. fyi @tajila @0xdaryl

openj9/jcl/src/java.base/share/classes/java/lang/Access.java

Lines 586 to 592 in 55c52ff

public void parkVirtualThread() {
if (Thread.currentThread() instanceof BaseVirtualThread bvt) {
bvt.park();
} else {
throw new WrongThreadException();
}
}

jdmpview output

!stack 0x3fee4064c00	!j9vmthread 0x3fee4064c00	!j9thread 0x3fee0022e18	tid 0x1db14 (121620) // (worker-1)

> info thread 121620
 process id: 121548

 no native threads found with specified id

Java threads not associated with known native threads:

    name:          <corrupt data encountered>
    id:            121620
    Thread object: <corrupt data encountered>
    Priority:      <corrupt data encountered>
    Thread.State:  WAITING
    JVMTI state:   ALIVE WAITING WAITING_INDEFINITELY IN_OBJECT_WAIT
      waiting to be notified on: "Live virtual thread list mutex" with ID 0x3ff9008e838 owner name: <unowned>
    Java stack frames:
     bp: 0x000003fed800a528  method: void java/lang/VirtualThread.notifyJvmtiUnmountBegin(boolean)  (Native Method)
      objects: 0x3ff8e30d2a0
     bp: 0x000003fed800a560  method: boolean java/lang/VirtualThread.yieldContinuation()  source: VirtualThread.java:372
      objects: 0x3ff8e30d2a0
     bp: 0x000003fed800a5c0  method: void java/lang/VirtualThread.parkNanos(long)  source: VirtualThread.java:537
      objects: 0x3ff8e30d2a0 0x3ff8e30d4d8
     bp: 0x000003fed800a600  method: void java/lang/Access.parkVirtualThread(long)  source: Access.java:502
      objects: <no objects in this frame>
     bp: 0x000003fed800a628  method: void jdk/internal/misc/VirtualThreads.park(long)  source: VirtualThreads.java:67
      objects: <no objects in this frame>
     bp: 0x000003fed800a650  method: void java/util/concurrent/locks/LockSupport.parkNanos(long)  source: LockSupport.java:408
      objects: <no objects in this frame>
     bp: 0x000003fed800a688  method: void GetStackTraceALot.lambda$main$0(java.util.concurrent.atomic.AtomicInteger, int)  source: GetStackTraceALot.java:93
      objects: 0x3ff8e30d3a8
     bp: 0x000003fed800a6a8  method: void GetStackTraceALot$$Lambda$17/0x0000000095261c60.run()  source: null:<data unavailable>
      objects: <no objects in this frame>
     bp: 0x000003fed800a708  method: void java/lang/VirtualThread.run(Runnable)  source: VirtualThread.java:292
      objects: 0x3ff8e30d2a0
     bp: 0x000003fed800a730  method: void java/lang/VirtualThread$VThreadContinuation.lambda$new$0(VirtualThread, Runnable)  source: VirtualThread.java:179
      objects: <no objects in this frame>
     bp: 0x000003fed800a750  method: void java/lang/VirtualThread$VThreadContinuation$$Lambda$23/0x0000000094ea1a08.run()  source: null:<data unavailable>
      objects: <no objects in this frame>
     bp: 0x000003fed800a778  method: void jdk/internal/vm/Continuation.execute(jdk.internal.vm.Continuation)  source: Continuation.java:171
      objects: 0x3ff8e30d538

> !stackslots 0x3fee4064c00
<3fee4064c00> *** BEGIN STACK WALK, flags = 00400001 walkThread = 0x000003FEE4064C00 ***
<3fee4064c00> 	ITERATE_O_SLOTS
<3fee4064c00> 	RECORD_BYTECODE_PC_OFFSET
<3fee4064c00> Initial values: walkSP = 0x000003FED800A4F8, PC = 0x0000000000000007, literals = 0x0000000000000000, A0 = 0x000003FED800A528, j2iFrame = 0x0000000000000000, ELS = 0x000003FF8E67EA10, decomp = 0x0000000000000000
<3fee4064c00> JNI native method frame: bp = 0x000003FED800A518, sp = 0x000003FED800A4F8, pc = 0x0000000000000007, cp = 0x0000000000000000, arg0EA = 0x000003FED800A528, flags = 0x0000000000000000
<3fee4064c00> 	Method: java/lang/VirtualThread.notifyJvmtiUnmountBegin(Z)V !j9method 0x000003FF902D6FA0
<3fee4064c00> 	Using signature mapper
<3fee4064c00> 	Arguments starting at 0x000003FED800A528 for UDATA(0x0000000000000002) slots
<3fee4064c00> 		O-Slot: a0[0x000003FED800A528] = 0x000003FF8E30D2A0   <---- CORRUPT VirtualThread 
<3fee4064c00> 		I-Slot: a1[0x000003FED800A520] = 0x0000000000000000
<3fee4064c00> Bytecode frame: bp = 0x000003FED800A540, sp = 0x000003FED800A530, pc = 0x000003FF1D4EB36A, cp = 0x000003FF902D5910, arg0EA = 0x000003FED800A560, flags = 0x0000000000000000
<3fee4064c00> 	Method: java/lang/VirtualThread.yieldContinuation()Z !j9method 0x000003FF902D6B00
<3fee4064c00> 	Bytecode index = 10
<3fee4064c00> 	Using local mapper
<3fee4064c00> 	Locals starting at 0x000003FED800A560 for 0x0000000000000004 slots
<3fee4064c00> 		O-Slot: a0[0x000003FED800A560] = 0x000003FF8E30D2A0   <---- CORRUPT VirtualThread 
<3fee4064c00> 		I-Slot: t1[0x000003FED800A558] = 0x00000001D800A562
<3fee4064c00> 		I-Slot: t2[0x000003FED800A550] = 0x000003FF1D4EB6FC
<3fee4064c00> 		I-Slot: t3[0x000003FED800A548] = 0x000003FF902D6BE0
<3fee4064c00> Bytecode frame: bp = 0x000003FED800A578, sp = 0x000003FED800A568, pc = 0x000003FF1D4EB6FC, cp = 0x000003FF902D5910, arg0EA = 0x000003FED800A5C0, flags = 0x0000000000000000
<3fee4064c00> 	Method: java/lang/VirtualThread.parkNanos(J)V !j9method 0x000003FF902D6BE0
<3fee4064c00> 	Bytecode index = 60
<3fee4064c00> 	Using local mapper
<3fee4064c00> 	Locals starting at 0x000003FED800A5C0 for 0x0000000000000009 slots
<3fee4064c00> 		O-Slot: a0[0x000003FED800A5C0] = 0x000003FF8E30D2A0   <---- CORRUPT VirtualThread 
<3fee4064c00> 		I-Slot: a1[0x000003FED800A5B8] = 0x000003FEE404FBD2
<3fee4064c00> 		I-Slot: a2[0x000003FED800A5B0] = 0x000000000007A120
<3fee4064c00> 		I-Slot: t3[0x000003FED800A5A8] = 0x000003FEE404FBD2
<3fee4064c00> 		I-Slot: t4[0x000003FED800A5A0] = 0x1729C1C46038E12E
<3fee4064c00> 		I-Slot: t5[0x000003FED800A598] = 0x000003FF902FADB0
<3fee4064c00> 		O-Slot: t6[0x000003FED800A590] = 0x000003FF8E30D4D8
<3fee4064c00> 		I-Slot: t7[0x000003FED800A588] = 0x000003FF1D4A1501
<3fee4064c00> 		I-Slot: t8[0x000003FED800A580] = 0x000003FF902A9200
<3fee4064c00> Bytecode frame: bp = 0x000003FED800A5D8, sp = 0x000003FED800A5C8, pc = 0x000003FF1D5248A1, cp = 0x000003FF902FAEB0, arg0EA = 0x000003FED800A600, flags = 0x0000000000000000
<3fee4064c00> 	Method: java/lang/Access.parkVirtualThread(J)V !j9method 0x000003FF902FADB0
<3fee4064c00> 	Bytecode index = 21
<3fee4064c00> 	Using local mapper
<3fee4064c00> 	Locals starting at 0x000003FED800A600 for 0x0000000000000005 slots
<3fee4064c00> 		I-Slot: a0[0x000003FED800A600] = 0x000003FF6E5F3890
<3fee4064c00> 		I-Slot: a1[0x000003FED800A5F8] = 0x000003FF1DEF4BA4
<3fee4064c00> 		I-Slot: a2[0x000003FED800A5F0] = 0x000000000007A120
<3fee4064c00> 		I-Slot: t3[0x000003FED800A5E8] = 0x000003FF8E443B68
<3fee4064c00> 		I-Slot: t4[0x000003FED800A5E0] = 0x000003FF8E443B68
<3fee4064c00> Bytecode frame: bp = 0x000003FED800A618, sp = 0x000003FED800A608, pc = 0x000003FF1DEF4BAA, cp = 0x000003FEE4057FC0, arg0EA = 0x000003FED800A628, flags = 0x0000000000000000
<3fee4064c00> 	Method: jdk/internal/misc/VirtualThreads.park(J)V !j9method 0x000003FEE4057D30
<3fee4064c00> 	Bytecode index = 6
<3fee4064c00> 	Using local mapper
<3fee4064c00> 	Locals starting at 0x000003FED800A628 for 0x0000000000000002 slots
<3fee4064c00> 		I-Slot: a0[0x000003FED800A628] = 0x0000000100000008
<3fee4064c00> 		I-Slot: a1[0x000003FED800A620] = 0x000000000007A120
<3fee4064c00> Bytecode frame: bp = 0x000003FED800A640, sp = 0x000003FED800A630, pc = 0x000003FF1D4E2430, cp = 0x000003FF902CEA70, arg0EA = 0x000003FED800A650, flags = 0x0000000000000000
<3fee4064c00> 	Method: java/util/concurrent/locks/LockSupport.parkNanos(J)V !j9method 0x000003FF902CE9E8
<3fee4064c00> 	Bytecode index = 16
<3fee4064c00> 	Using local mapper
<3fee4064c00> 	Locals starting at 0x000003FED800A650 for 0x0000000000000002 slots
<3fee4064c00> 		I-Slot: a0[0x000003FED800A650] = 0x00000001E404FC58
<3fee4064c00> 		I-Slot: a1[0x000003FED800A648] = 0x000000000007A120

> !j9object 0x000003FF8E30D2A0   <---- CORRUPT VirtualThread 
Unable to read object clazz at 0x000003FF8E30D2A0 (clazz = 0x000003FF6E777300)

> !j9x 0x000003FF8E30D2A0-0x50,0x100
0x3FF8E30D250 :  000003ff6e65f62c 0000000000000000 [ ....ne.,........ ]
0x3FF8E30D260 :  000003fed404e0d0 000003fed4088398 [ ................ ]
0x3FF8E30D270 :  000003ff6e749640 000003ff8e310570 [ ....nt.@.....1.p ]
0x3FF8E30D280 :  000003ff8e30d218 0000000000000000 [ .....0.......... ]
0x3FF8E30D290 :  0000000000000000 0601100a00000000 [ ................ ]
0x3FF8E30D2A0 :  000003ff6e7773ec 0000000000000000 [ ....nws......... ] <--- CORRUPT VirtualThread
0x3FF8E30D2B0 :  0000000000000000 000000000000001c [ ................ ]
0x3FF8E30D2C0 :  0000000000000000 0000000000000000 [ ................ ]
0x3FF8E30D2D0 :  000003ff6e58b308 000003ff6e5f2d20 [ ....nX......n_-  ]
0x3FF8E30D2E0 :  000003ff6e6ca908 0000000000000000 [ ....nl.......... ]
0x3FF8E30D2F0 :  0000000000000000 0000000000000000 [ ................ ]
0x3FF8E30D300 :  0000000000000000 000003ff8e310590 [ .............1.. ]
0x3FF8E30D310 :  0000000000000000 0000000000000000 [ ................ ]
0x3FF8E30D320 :  0000000000000000 0000000000000000 [ ................ ]
0x3FF8E30D330 :  000003ff8e3001f0 0000000000000000 [ .....0.......... ]
0x3FF8E30D340 :  0000000000000000 0000000000000000 [ ................ ]

@JasonFengJ9
Copy link
Member Author

Results from 50x grinder -Xtrace:maximal={j9scar{level5}},buffers=2m:
Iteration 0

12:06:44  17:06:34.402 0x70ba3800a400    j9mm.141    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/build-scripts/jobs/jdk19/jdk19-linux-ppc64le-openj9/workspace/build/src/omr/gc/base/standard/Scavenger.cpp:1657: ((false))

Iteration 1 - timed out
Iteration 2 - passed
Iteration 3 - passed
Iteration 4 - no failure though the job status UNSTABLE

@tajila
Copy link
Contributor

tajila commented Nov 28, 2022 •
edited
Loading

50x grinder with assertions https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/29825

@babsingh
Copy link
Contributor

https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/29799/

@tajila Unzipping files failed in the grinder.

@pshipton
Copy link
Member

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_mac_Nightly/62/ - no diagnostics

@dmitripivkine
Copy link
Contributor

dmitripivkine commented Nov 30, 2022 •
edited
Loading

another failure #16249 (comment) (single element Virtual Threads List next/previous stale pointer)

@pshipton
Copy link
Member

pshipton commented Dec 1, 2022 •
edited
Loading

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_ppc64_aix_Nightly/61/
jdk_lang_1
java/lang/Thread/virtual/ParkWithFixedThreadPool.java

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_sanity.openjdk_ppc64_aix_Nightly/61/openjdk_test_output.tar.gz

00:16:44  Type=Segmentation error vmState=0x0002000f
00:16:44  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000032
00:16:44  Handler1=09001000A1017FB0 Handler2=09001000A0F57020
00:16:44  R0=321825CB00000000 R1=0000010022A01760 R2=09001000A0F26978 R3=0000010010209150
00:16:44  R4=0000010022A0B3D8 R5=0000010022A01B10 R6=0000010022A018B0 R7=0000010022A018A8
00:16:44  R8=0000010022A018A0 R9=000001001038F450 R10=0000000000000000 R11=0A0000001FF84A80
00:16:44  R12=FFFFFFFF82000022 R13=0000010022A0C800 R14=0000000000000001 R15=0A0000001FD90E48
00:16:44  R16=0A0000001FF9A390 R17=0A0000001FF9A390 R18=000001001038F370 R19=0000000000000000
00:16:44  R20=0000000000000000 R21=0A0000001FD91CB0 R22=000001001038F348 R23=0000000000002C20
00:16:44  R24=0000000000002C18 R25=00000100102091D8 R26=321825CB00000000 R27=0000000000000000
00:16:44  R28=0000010022A018A0 R29=0000010022A018A8 R30=0000010022A018B0 R31=0000010022A01B10
00:16:44  IAR=090000001033D364 LR=09000000104038B8 MSR=A00000000200D032 CTR=090000001042C424
00:16:44  CR=2400002200000004 FPSCR=8202000000000000 XER=0000000482020000
00:16:44  FPR0 000000000000004c (f: 76.000000, d: 3.754899e-322)
00:16:44  FPR1 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
00:16:44  FPR2 3fc7344ba27d52af (f: 2726122240.000000, d: 1.812834e-01)
00:16:44  FPR3 3f51261904000000 (f: 67108864.000000, d: 1.046681e-03)
00:16:44  FPR4 bf90ecf58ddc7f4a (f: 2380037888.000000, d: -1.652893e-02)
00:16:44  FPR5 40a6d60000000000 (f: 0.000000, d: 2.923000e+03)
00:16:44  FPR6 4070b1f20d4afc0c (f: 223018000.000000, d: 2.671216e+02)
00:16:44  FPR7 bfb62e42fefa4000 (f: 4277813248.000000, d: -8.664340e-02)
00:16:44  FPR8 3f6069c0ff9e20ca (f: 4288553216.000000, d: 2.003552e-03)
00:16:44  FPR9 402bb9d3beb8d000 (f: 3199782912.000000, d: 1.386294e+01)
00:16:44  FPR10 408f400000000000 (f: 0.000000, d: 1.000000e+03)
00:16:44  FPR11 43300000000003e8 (f: 1000.000000, d: 4.503600e+15)
00:16:44  FPR12 4530000000000000 (f: 0.000000, d: 1.934281e+25)
00:16:44  FPR13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  Module=/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_ppc64_aix_Nightly/openjdkbinary/j2sdk-image/lib/default/libj9gc_full29.so
00:16:44  Module_base_address=09000000102AF000
00:16:44  Target=2_90_20221201_130 (AIX 7.1)
00:16:44  CPU=ppc64 (16 logical CPUs) (0x780000000 RAM)
00:16:44  ----------- Stack Backtrace -----------
00:16:44  _ZN12MM_Scavenger14copyForVariantILb0EEEP8J9ObjectP22MM_EnvironmentStandardP18MM_ForwardedHeader+0xb4 (0x09000000104038B8 [libj9gc_full29.so+0x1548b8])
00:16:44  _ZN12MM_Scavenger26incrementalScanCacheBySlotEP22MM_EnvironmentStandardP24MM_CopyScanCacheStandard+0x1170 (0x090000001040DCF4 [libj9gc_full29.so+0x15ecf4])
00:16:44  _ZN12MM_Scavenger12completeScanEP22MM_EnvironmentStandard+0x2f0 (0x090000001040EAB4 [libj9gc_full29.so+0x15fab4])
00:16:44  _ZN12MM_Scavenger24workThreadGarbageCollectEP22MM_EnvironmentStandard+0x388 (0x090000001040EF4C [libj9gc_full29.so+0x15ff4c])
00:16:44  _ZN23MM_ParallelScavengeTask3runEP18MM_EnvironmentBase+0x10 (0x0900000010444694 [libj9gc_full29.so+0x195694])
00:16:44  _ZN21MM_ParallelDispatcher16workerEntryPointEP18MM_EnvironmentBase+0x1d4 (0x090000001049DAD8 [libj9gc_full29.so+0x1eead8])
00:16:44  _Z23dispatcher_thread_proc2P14OMRPortLibraryPv+0x1d8 (0x090000001049D7DC [libj9gc_full29.so+0x1ee7dc])
00:16:44  omrsig_protect+0x4fc (0x090000000E24E1E0 [libj9prt29.so+0x5f1e0])
00:16:44  dispatcher_thread_proc+0x50 (0x090000001049D8B4 [libj9gc_full29.so+0x1ee8b4])
00:16:44  thread_wrapper+0x558 (0x090000000E2B799C [libj9thr29.so+0x599c])
00:16:44  _pthread_body+0xf0 (0x090000000051FE14 [libpthreads.a+0x3e14])
00:16:44  ---------------------------------------

@pshipton
Copy link
Member

pshipton commented Dec 2, 2022

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_Nightly/61
jdk_lang_1
java/lang/Thread/virtual/stress/GetStackTraceALot.java#id0
https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_Nightly/61/openjdk_test_output.tar.gz

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_x86-64_mac_Nightly/60
jdk_lang_1
java/lang/Thread/virtual/ParkWithFixedThreadPool.java
No diagnostics.

@tajila tajila mentioned this issue Dec 4, 2022
Merged
tajila added a commit to tajila/openj9 that referenced this issue Dec 5, 2022
@tajila
Fix up potential stale pointer after GC
ef7a4c8 
Fixes eclipse-openj9#16351

Signed-off-by: Tobi Ajila <atobia@ca.ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
comp:vm project:loom Used to track Project Loom related work test failure triageRequired
Projects
None yet
Milestone
Java 19 0.37
Development

Successfully merging a pull request may close this issue.

Fix up potential stale pointer after GC tajila/openj9
5 participants
@babsingh @tajila @JasonFengJ9 @pshipton @dmitripivkine