Refactor VirtualThread synchronization design #16855

fengxue-IS · 2023-03-08T20:02:58Z

Removes use of liveVirtualThreadList/liveVirtualThreadListMutex in the VM and rely on GC's Continuation list.
This design uses an atomic based RW lock with busy waiting for vthread access during mounted/unmount transition.
Global access in JVMTI is a slow path and done through Exclusive VM Access which doesn't require additional locking.

mutual dependency:
jdk19: ibmruntimes/openj9-openjdk-jdk19#83
jdk20: ibmruntimes/openj9-openjdk-jdk20#32
jdknext: ibmruntimes/openj9-openjdk-jdk#574

Fixes: #16728

Signed-off-by: Jack Lu Jack.S.Lu@ibm.com

fengxue-IS · 2023-03-08T20:05:14Z

FYI @babsingh @tajila @gacholio

runtime/gc_base/GCExtensions.hpp

fengxue-IS · 2023-03-10T06:29:25Z

Update

Per discussion with @babsingh, will merge design concept from JVMTI suspend and resume rework to remove the virtual thread list #16846 to use EVMA instead of a RW-lock for controlling the global vthread access. This should completely remove the mount/unmount contention seen in stress testing.
Refactor getStackTraceImpl to redo the check after calling haltThreadForInspection so it will not require using VirtualThread.inspectorCount for sync
Dynamically set VirtualThread.notifyJvmtiEvents to remove all dependency on the notifyJvmtiMount/Unmount[Begin/End] API if JVMTI is not used.

gacholio · 2023-03-10T17:45:32Z

use EVMA instead of a RW-lock for controlling the global vthread access

That's fine for the JVMTI code but will probably be unuseably slow for any hot code path. EVMA is an extremely expensive operation.

fengxue-IS · 2023-03-10T18:52:25Z

use EVMA instead of a RW-lock for controlling the global vthread access

That's fine for the JVMTI code but will probably be unuseably slow for any hot code path. EVMA is an extremely expensive operation.

Currently JVMTI suspend/resume all is the only case where the RW lock is using writer access, so I assume by switching to EMVA will just remove the need for any locking for non-jvmti path

gacholio · 2023-03-14T15:58:17Z

Have you measured CPU usage before and after this change? Spinning is OK for short-lived operations, but I suspect the CPU usage will spike heavily with this solution.

runtime/oti/VMHelpers.hpp

runtime/oti/vm_api.h

fengxue-IS · 2023-03-15T21:01:54Z

updated to the EVMA design, perf result 7x better on nocompressedrefs, 20% perf better on compressedrefs

runtime/vm/ContinuationHelpers.cpp

runtime/compiler/control/HookedByTheJit.cpp

…ed thread Signed-off-by: Jack Lu <Jack.S.Lu@ibm.com>

Signed-off-by: Jack Lu <Jack.S.Lu@ibm.com>

fengxue-IS · 2023-03-29T20:35:55Z

addressed all review comments, locally compile/tested against JTReg stress test on x86 linux.

runtime/oti/VMHelpers.hpp

tajila · 2023-03-29T22:37:26Z

jenkins test sanity.functional,extended.functional,sanity.openjdk,sanity.system,extended.system xlinux jdk19

fengxue-IS · 2023-03-30T04:49:48Z

test failure due to assertion added in #16713

j9vm.224    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK19_x86-64_linux_Personal/openj9/runtime/vm/ContinuationHelpers.cpp:264: ((VM_VMHelpers::isFinished(continuation->state)))

This code is incorrect since when a yielded continuation gets freed by GC due to being unreachable, this assertion will fail (though state actually correctly reflected). This assertion made the assumption that unfinished vthread object will be kept alive by the VM's liveVirtualThreadList even if it is no longer reachable from user code which is no longer the case. FYI @LinHu2016

PS: I think the only thing we can assert during freeContinuation is that the continuation cannot be mounted (otherwise we are deleting a carrierThread's data)

fengxue-IS · 2023-03-30T05:19:07Z

jenkins test sanity.functional,extended.functional,sanity.openjdk,sanity.system,extended.system xlinux jdk19

PR testing would need to depends on extension repo changes (something like)

jenkins test sanity.functional,extended.functional,sanity.openjdk,sanity.system,extended.system xlinux jdk19 depends ibmruntimes/openj9-openjdk-jdk19#83

gacholio · 2023-03-30T13:42:26Z

@fengxue-IS Are you going to fix the assertion here?

Is there any reason ibmruntimes/openj9-openjdk-jdk19#83 can't be merged?

fengxue-IS · 2023-03-30T14:21:09Z

@fengxue-IS Are you going to fix the assertion here?

I can fix the assertion in this pr, just want to confirm with lin that i correctly understand the reason for having the assertion originally.

Is there any reason ibmruntimes/openj9-openjdk-jdk19#83 can't be merged?

the extension change is co-dependent on this pr (continuation.vthread field is added in this change)

fengxue-IS · 2023-03-30T18:00:20Z

pushing fix for assertion, PR test link for reference:
https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_x86-64_linux_Personal/9/
https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.functional_x86-64_linux_Personal/88/
https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.functional_x86-64_linux_Personal/3/
https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.system_x86-64_linux_Personal/2/
https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.system_x86-64_linux_Personal/1/

Continuation must be unmounted/not being scanned when free. Signed-off-by: Jack Lu <Jack.S.Lu@ibm.com>

gacholio · 2023-03-30T18:07:37Z

jenkins test sanity.functional,extended.functional,sanity.openjdk,sanity.system,extended.system xlinux jdk19 depends ibmruntimes/openj9-openjdk-jdk19#83

gacholio · 2023-03-30T18:23:36Z

I won't merge this if the testing passes due to the dependency. Someone who is a committer in both repos will have to do it.

tajila · 2023-03-30T18:26:17Z

Sure Ill do it

fengxue-IS · 2023-03-31T03:04:01Z

build failed due to javadoc compile error, pending fix #17077

gacholio · 2023-03-31T12:14:47Z

jenkins test sanity.functional,extended.functional,sanity.openjdk,sanity.system,extended.system xlinux jdk19 depends ibmruntimes/openj9-openjdk-jdk19#83

runtime/j9vm/javanextvmi.cpp

keithc-ca · 2023-03-31T19:43:00Z

runtime/oti/j9nonbuilder.h

-	j9object_t *liveVirtualThreadList;
-	omrthread_monitor_t liveVirtualThreadListMutex;
-	volatile BOOLEAN inspectingLiveVirtualThreadList;
-	UDATA virtualThreadLinkNextOffset;


I don't see an update to DDR code to deal with the removal of virtualThreadLinkNextOffset; as a result, the !vthreads command is now broken: #17084 tracks that problem.

VThreadList lock contention fixed by eclipse-openj9/openj9#16855 Signed-off-by: Jack Lu <Jack.S.Lu@ibm.com>

fengxue-IS added project:loom Used to track Project Loom related work comp:vm perf labels Mar 8, 2023

tajila requested review from babsingh, tajila and gacholio March 8, 2023 20:21

tajila added the jdk19 label Mar 8, 2023

fengxue-IS force-pushed the remove_vthreadlist branch 3 times, most recently from 8e773b1 to 91fb21f Compare March 9, 2023 18:29

babsingh mentioned this pull request Mar 9, 2023

JVMTI suspend and resume rework to remove the virtual thread list #16846

Closed

fengxue-IS force-pushed the remove_vthreadlist branch from 91fb21f to efdec55 Compare March 9, 2023 19:24

amicic reviewed Mar 9, 2023

View reviewed changes

runtime/gc_base/GCExtensions.hpp Show resolved Hide resolved

fengxue-IS force-pushed the remove_vthreadlist branch from efdec55 to 3f066c7 Compare March 9, 2023 20:25

fengxue-IS force-pushed the remove_vthreadlist branch from 3f066c7 to a7ac1a1 Compare March 13, 2023 15:56

thallium mentioned this pull request Mar 14, 2023

JVMTI SetGetThreadLocalStorageStressTest failed with unexpected value in storage. #16687

Closed

tajila reviewed Mar 14, 2023

View reviewed changes

runtime/oti/VMHelpers.hpp Outdated Show resolved Hide resolved

tajila reviewed Mar 14, 2023

View reviewed changes

runtime/oti/vm_api.h Show resolved Hide resolved

fengxue-IS force-pushed the remove_vthreadlist branch from a7ac1a1 to c934c05 Compare March 15, 2023 18:18

fengxue-IS force-pushed the remove_vthreadlist branch from 008bd24 to af3fa02 Compare March 15, 2023 22:17

gacholio reviewed Mar 20, 2023

View reviewed changes

runtime/vm/ContinuationHelpers.cpp Outdated Show resolved Hide resolved

gacholio reviewed Mar 20, 2023

View reviewed changes

runtime/compiler/control/HookedByTheJit.cpp Show resolved Hide resolved

fengxue-IS force-pushed the remove_vthreadlist branch from ef82390 to 52dddc4 Compare March 21, 2023 01:35

fengxue-IS mentioned this pull request Mar 21, 2023

Initialize Continuation.vthread in constructor ibmruntimes/openj9-openjdk-jdk19#83

Merged

fengxue-IS added 3 commits March 29, 2023 16:34

Fix vthread suspend order to avoid dispatching mount hook for suspend…

cf7e46f

…ed thread Signed-off-by: Jack Lu <Jack.S.Lu@ibm.com>

Change GC continuation list option default to enable

14084ce

Signed-off-by: Jack Lu <Jack.S.Lu@ibm.com>

Forces cache flush before walking Continuation list

7873bf8

Signed-off-by: Jack Lu <Jack.S.Lu@ibm.com>

fengxue-IS force-pushed the remove_vthreadlist branch from 3fe37ae to 7873bf8 Compare March 29, 2023 20:35

tajila approved these changes Mar 29, 2023

View reviewed changes

runtime/oti/VMHelpers.hpp Show resolved Hide resolved

Refactor freeContinuation assert

c50c552

Continuation must be unmounted/not being scanned when free. Signed-off-by: Jack Lu <Jack.S.Lu@ibm.com>

fengxue-IS force-pushed the remove_vthreadlist branch from 5801f93 to c50c552 Compare March 30, 2023 18:00

gacholio approved these changes Mar 30, 2023

View reviewed changes

gacholio reviewed Mar 31, 2023

View reviewed changes

runtime/j9vm/javanextvmi.cpp Show resolved Hide resolved

tajila merged commit 55b9267 into eclipse-openj9:master Mar 31, 2023

keithc-ca mentioned this pull request Mar 31, 2023

DDR vthreads command no longer works #17084

Closed

keithc-ca reviewed Mar 31, 2023

View reviewed changes

This was referenced Mar 31, 2023

Correctly ifdef Java19 code #17086

Merged

(0.37) Refactor VirtualThread synchronization design #17094

Merged

fengxue-IS added a commit to fengxue-IS/aqa-tests that referenced this pull request Apr 3, 2023

Re-enable Skynet test

c29075e

VThreadList lock contention fixed by eclipse-openj9/openj9#16855 Signed-off-by: Jack Lu <Jack.S.Lu@ibm.com>

fengxue-IS mentioned this pull request Apr 3, 2023

Re-enable Skynet test adoptium/aqa-tests#4491

Merged

renfeiw pushed a commit to adoptium/aqa-tests that referenced this pull request Apr 3, 2023

Re-enable Skynet test (#4491)

ed600fc

VThreadList lock contention fixed by eclipse-openj9/openj9#16855 Signed-off-by: Jack Lu <Jack.S.Lu@ibm.com>

fengxue-IS mentioned this pull request Apr 4, 2023

jdk19 OpenJDK java/lang/Thread/virtual/stress/Skynet.java hang / timeout #16728

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor VirtualThread synchronization design #16855

Refactor VirtualThread synchronization design #16855

fengxue-IS commented Mar 8, 2023 •

edited

Loading

fengxue-IS commented Mar 8, 2023

fengxue-IS commented Mar 10, 2023

gacholio commented Mar 10, 2023

fengxue-IS commented Mar 10, 2023

gacholio commented Mar 14, 2023

fengxue-IS commented Mar 15, 2023

fengxue-IS commented Mar 29, 2023

tajila commented Mar 29, 2023

fengxue-IS commented Mar 30, 2023 •

edited

Loading

fengxue-IS commented Mar 30, 2023

gacholio commented Mar 30, 2023

fengxue-IS commented Mar 30, 2023

fengxue-IS commented Mar 30, 2023

gacholio commented Mar 30, 2023

gacholio commented Mar 30, 2023

tajila commented Mar 30, 2023

fengxue-IS commented Mar 31, 2023

gacholio commented Mar 31, 2023

keithc-ca Mar 31, 2023

Refactor VirtualThread synchronization design #16855

Refactor VirtualThread synchronization design #16855

Conversation

fengxue-IS commented Mar 8, 2023 • edited Loading

fengxue-IS commented Mar 8, 2023

fengxue-IS commented Mar 10, 2023

gacholio commented Mar 10, 2023

fengxue-IS commented Mar 10, 2023

gacholio commented Mar 14, 2023

fengxue-IS commented Mar 15, 2023

fengxue-IS commented Mar 29, 2023

tajila commented Mar 29, 2023

fengxue-IS commented Mar 30, 2023 • edited Loading

fengxue-IS commented Mar 30, 2023

gacholio commented Mar 30, 2023

fengxue-IS commented Mar 30, 2023

fengxue-IS commented Mar 30, 2023

gacholio commented Mar 30, 2023

gacholio commented Mar 30, 2023

tajila commented Mar 30, 2023

fengxue-IS commented Mar 31, 2023

gacholio commented Mar 31, 2023

keithc-ca Mar 31, 2023

Choose a reason for hiding this comment

fengxue-IS commented Mar 8, 2023 •

edited

Loading

fengxue-IS commented Mar 30, 2023 •

edited

Loading