Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests segfault in Mono #81249

danmoseley · 2023-01-27T00:44:09Z

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-81202-merge-135c923415104d6c85/Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests/1/console.6c3784b3.log?helixlogtype=result

net8.0-linux-Debug-arm64-Mono_release-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8

Possible stack trace for the crash

Thread 9 (Thread 0xff7909f741c0 (LWP 34)):
#0  0x0000ff791595bd5c in __waitpid (pid=<optimized out>, stat_loc=0xff7909f6e9f0, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x0000ff79153519e8 in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:843
#2  mono_dump_native_crash_info (signal=<optimized out>, mctx=0xff7909f6f450, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:870
#3  0x0000ff79153105c0 in mono_handle_native_crash (signal=0xff791510e1d2 "SIGSEGV", mctx=0xff7909f6f450, info=0xff7909f6f7b0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3005
#4  0x0000ff7915279b90 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0xff7909f6f7b0, context=0xff7909f6f830, debug_fault_addr=0x0) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3749
#5  <signal handler called>
#6  0x0000000000000000 in ?? ()
#7  0x0000ff7908de1d18 in ?? ()
#8  0x0000ff7914ad9848 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

(original issue description note: this is not the problematic stack trace)

Not sure how it indicates which thread failed, but I'm guessing it's this one?

Thread 10 (Thread 0xff7909d431c0 (LWP 35)):
#0  monoeg_g_hash_table_insert_replace (hash=0xab9b66ab1fb0, key=0xff78e945dca0, value=0xff78e945dca0, replace=0) at /__w/1/s/src/mono/mono/eglib/ghashtable.c:246
#1  0x0000ff79151467b4 in mono_class_inflate_generic_method_full_checked (method=<optimized out>, klass_hint=<optimized out>, context=0xff78e8e928f8, error=<optimized out>) at /__w/1/s/src/mono/mono/metadata/class.c:1312
#2  0x0000ff79151588f0 in mono_class_setup_vtable_ginst (klass=0xff78e8e92920, in_setup=<optimized out>) at /__w/1/s/src/mono/mono/metadata/class-setup-vtable.c:1363
#3  mono_class_setup_vtable_general (klass=0xff78e8e92920, overrides=<optimized out>, onum=<optimized out>, in_setup=<optimized out>) at /__w/1/s/src/mono/mono/metadata/class-setup-vtable.c:1723
#4  0x0000ff7915158484 in mono_class_setup_vtable_full (klass=0xff78e8e92920, in_setup=0xff78e945a130) at /__w/1/s/src/mono/mono/metadata/class-setup-vtable.c:960
#5  0x0000ff791529ca68 in mono_method_to_ir (cfg=<optimized out>, method=<optimized out>, start_bblock=0xff78e9443078, end_bblock=0xff78e94431b8, return_var=0x0, inline_args=<optimized out>, inline_offset=<optimized out>, is_virtual_call=<optimized out>) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:7924
#6  0x0000ff7915271728 in mini_method_compile (method=<optimized out>, opts=374417919, flags=JIT_FLAG_RUN_CCTORS, parts=0, aot_method_index=-1) at /__w/1/s/src/mono/mono/mini/mini.c:3495
#7  0x0000ff791527412c in mono_jit_compile_method_inner (method=0xff78e8e89940, opt=-381297504, error=0xff7909d3c970) at /__w/1/s/src/mono/mono/mini/mini.c:4129
#8  0x0000ff7915278cb0 in mono_jit_compile_method_with_opt (method=0xff78e8e89940, opt=374417919, jit_only=0, error=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2709
#9  jit_compile_method_with_opt_cb (arg=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2764
#10 jit_compile_method_with_opt (params=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2780
#11 0x0000ff79152780dc in mono_jit_compile_method (method=<optimized out>, error=0xff7909d3c970) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2799
#12 0x0000ff7915312a2c in common_call_trampoline (regs=0xff7909d3ca00, code=0xff79082dc1ec "", m=0xff78e8e89940, vt=0x0, vtable_slot=<optimized out>, error=0xff7909d3c970) at /__w/1/s/src/mono/mono/mini/mini-trampolines.c:618
#13 0x0000ff791531254c in mono_magic_trampoline (regs=0xff7909d3ca00, code=0xff79082dc1ec "", arg=0xff78e93e72a0, tramp=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-trampolines.c:759
#14 0x0000ff7914ee8304 in ?? ()
#15 0x0000ff7914bdf620 in ?? ()
#16 0x0000ff7914b219c0 in ?? ()
#17 0x0064002e00740073 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

there is a core dump
https://dev.azure.com/dnceng-public/public/_build/results?buildId=148909&view=ms.vss-test-web.build-test-results-tab&runId=3170018&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab&resultId=200309

The text was updated successfully, but these errors were encountered:

stephentoub · 2023-01-27T14:01:53Z

There's another issue representing a crash in source generators on mono that's been plaguing us in CI:
#81123
I don't know if it's the same thing.

danmoseley · 2023-02-07T00:32:23Z

@janvorli would you or someone on the VM team be interested in the dump here before it vanishes? at least #81123 seems fairly common and this might be related.

janvorli · 2023-02-07T00:48:31Z

@danmoseley this is mono, so it would be better to point some mono folks to this. The coreclr runtime team doesn't have experience debugging mono.

danmoseley · 2023-02-07T01:32:54Z

@janvorli - doh - of course, brainstorm. Thanks.

@SamMonoRT there's a dump here if you want it, not sure how long before they time out.

SamMonoRT · 2023-02-07T12:54:32Z

@lambdageek - can you take an initial look

lambdageek · 2023-02-07T14:04:22Z

Thread 10 looks ok, actually. it's just doing some work. Thread 9 looks like it crashed:

Thread 9 (Thread 0xff7909f741c0 (LWP 34)):
#0  0x0000ff791595bd5c in __waitpid (pid=<optimized out>, stat_loc=0xff7909f6e9f0, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x0000ff79153519e8 in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:843
#2  mono_dump_native_crash_info (signal=<optimized out>, mctx=0xff7909f6f450, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:870
#3  0x0000ff79153105c0 in mono_handle_native_crash (signal=0xff791510e1d2 "SIGSEGV", mctx=0xff7909f6f450, info=0xff7909f6f7b0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3005
#4  0x0000ff7915279b90 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0xff7909f6f7b0, context=0xff7909f6f830, debug_fault_addr=0x0) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3749
#5  <signal handler called>
#6  0x0000000000000000 in ?? ()
#7  0x0000ff7908de1d18 in ?? ()
#8  0x0000ff7914ad9848 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

and some output from mono_dump_native_crash_info

=====================================================
instruction pointer is NULL, skip dumping
=====================================================

Frames 7 and 8 look like JITed code. we'll need to try repro locally.

~~I'll try to grab the coredump once I'm on my work computer~~ Core dump expired.

steveisok · 2023-02-15T17:29:02Z

@lambdageek the links to builds in #81123 might contain coredumps.

lewing · 2023-02-15T17:45:44Z

@lambdageek as another data point the crashes only seem to happen on arm64

lambdageek · 2023-02-16T20:00:48Z

Got it to crash in a VM after a couple of hours of running the test in a loop. Not the same crash as this issue, unfortunately. An assertion failure during class setup due to - I'm guessing - a data race:

	g_assert (klass == klass->supertypes [klass->idepth - 1]);

I see two threads in mono_class_setup_vtable_full for the same class. It does seem like it's possible for klass->supertypes and klass->idepth not to be set atomically.

I can't convince myself that it would cause the same kinds of failures we see in the CI jobs.

lambdageek · 2023-02-17T17:26:14Z

Probably same as #81123 - which is a very similar stack trace in the Roslyn3.11 test suite. Going to try running that other testsuite in a loop, too

carlossanlop · 2023-03-15T23:45:15Z

Seen again in an unrelated PR in main: #83356

Queue: Libraries Test Run release mono linux arm64 Debug
Job: https://dev.azure.com/dnceng-public/public/_build/results?buildId=206231&view=logs&j=0aa53445-e474-5f87-17f4-cf1800f4293c&t=f8159c71-541f-59ca-1f6b-cfff92851d99
Log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-83356-merge-426b96be0b3a42819c/Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests/1/console.1a24fb92.log?helixlogtype=result

Callstacks (expand)

=================================================================
	External Debugger Dump:
=================================================================
[New LWP 23]
[New LWP 24]
[New LWP 25]
[New LWP 26]
[New LWP 29]
[New LWP 30]
[New LWP 31]
[New LWP 32]
[New LWP 33]
[New LWP 34]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
futex_wait_cancelable (private=0, expected=0, futex_word=0xab1a6c0e25ac) at ../sysdeps/nptl/futex-internal.h:186
186	../sysdeps/nptl/futex-internal.h: No such file or directory.
  Id   Target Id                                        Frame 
* 1    Thread 0xff0694dec010 (LWP 22) "dotnet"          futex_wait_cancelable (private=0, expected=0, futex_word=0xab1a6c0e25ac) at ../sysdeps/nptl/futex-internal.h:186
  2    Thread 0xff0693bff1b0 (LWP 23) "SGen worker"     futex_wait_cancelable (private=0, expected=0, futex_word=0xff069480fe80 <work_cond+40>) at ../sysdeps/nptl/futex-internal.h:186
  3    Thread 0xff0691e611b0 (LWP 24) ".NET EventPipe"  0x0000ff0694a30ef4 in __GI___poll (fds=0xff06840034f0, nfds=1, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  4    Thread 0xff0691c601b0 (LWP 25) "Finalizer"       futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0xff0694801950 <finalizer_sem>) at ../sysdeps/nptl/futex-internal.h:323
  5    Thread 0xff06898ff1b0 (LWP 26) ".NET SigHandler" __libc_read (nbytes=1, buf=0xff06898fe977, fd=<optimized out>) at ../sysdeps/unix/sysv/linux/read.c:26
  6    Thread 0xff06897d81b0 (LWP 29) ".NET Long Runni" futex_wait_cancelable (private=0, expected=0, futex_word=0xff06780057e8) at ../sysdeps/nptl/futex-internal.h:186
  7    Thread 0xff06895c71b0 (LWP 30) ".NET TP Worker"  futex_abstimed_wait_cancelable (private=0, abstime=0xff06895c60e8, clockid=<optimized out>, expected=0, futex_word=0xff067c005348) at ../sysdeps/nptl/futex-internal.h:323
  8    Thread 0xff06890cb1b0 (LWP 31) ".NET TP Gate"    futex_abstimed_wait_cancelable (private=0, abstime=0xff06890ca2a8, clockid=<optimized out>, expected=0, futex_word=0xff0670005488) at ../sysdeps/nptl/futex-internal.h:323
  9    Thread 0xff068906a1b0 (LWP 32) ".NET TP Worker"  futex_abstimed_wait_cancelable (private=0, abstime=0xff0689069320, clockid=<optimized out>, expected=0, futex_word=0xff0689069398) at ../sysdeps/nptl/futex-internal.h:323
  10   Thread 0xff0688e491b0 (LWP 33) ".NET Long Runni" mono_metadata_generic_inst_hash (data=0xff0688e419e0) at /__w/1/s/src/mono/mono/metadata/metadata.c:1878
  11   Thread 0xff0688c481b0 (LWP 34) ".NET Long Runni" 0x0000ff0694a0c77c in __GI___wait4 (pid=<optimized out>, stat_loc=0xff0688c41350, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27

Thread 11 (Thread 0xff0688c481b0 (LWP 34) ".NET Long Runni"):
#0  0x0000ff0694a0c77c in __GI___wait4 (pid=<optimized out>, stat_loc=0xff0688c41350, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27
#1  0x0000ff069460eb34 in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:843
#2  mono_dump_native_crash_info (signal=<optimized out>, mctx=0xff0688c41db0, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:870
#3  0x0000ff06945c5f54 in mono_handle_native_crash (signal=0xff06944df2e5 "SIGSEGV", mctx=0xff0688c41db0, info=0xff0688c42110) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2979
#4  0x0000ff069452da60 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0xff0688c42110, context=0xff0688c42190, debug_fault_addr=0x0) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3758
#5  <signal handler called>
#6  0x0000000000000000 in ?? ()
#7  0x0000ff068804d478 in ?? ()
#8  0x0000ff0693dc6ae0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 10 (Thread 0xff0688e491b0 (LWP 33) ".NET Long Runni"):
#0  mono_metadata_generic_inst_hash (data=0xff0688e419e0) at /__w/1/s/src/mono/mono/metadata/metadata.c:1878
#1  0x0000ff06946a28dc in monoeg_g_hash_table_lookup_extended (hash=0xab1a6b0dd880, key=0xff0688e419e0, orig_key=<optimized out>, value=<optimized out>) at /__w/1/s/src/mono/mono/eglib/ghashtable.c:338
#2  monoeg_g_hash_table_lookup (hash=0xab1a6b0dd880, key=0xff0688e419e0) at /__w/1/s/src/mono/mono/eglib/ghashtable.c:321
#3  0x0000ff0694724d8c in mono_metadata_get_canonical_generic_inst (candidate=0xff0688e419e0) at /__w/1/s/src/mono/mono/metadata/metadata.c:3414
#4  0x0000ff0694724cc4 in mono_metadata_get_generic_inst (type_argc=<optimized out>, type_argv=0xff066895f730) at /__w/1/s/src/mono/mono/metadata/metadata.c:3381
#5  0x0000ff0694725758 in mono_metadata_inflate_generic_inst (ginst=0xab1a6b1bb588, context=0xab1a6b1a7198, error=0xff066892f968) at /__w/1/s/src/mono/mono/metadata/metadata.c:3571
#6  0x0000ff069470ea74 in method_from_methodspec (image=0xab1a6b090b80, context=0xab1a6b1a7198, idx=<optimized out>, error=0xff066892f968) at /__w/1/s/src/mono/mono/metadata/loader.c:980
#7  mono_get_method_from_token (image=0xab1a6b090b80, token=721422599, klass=<optimized out>, context=0xab1a6b1a7198, used_context=0xff0688e41a9c, error=0xff066892f968) at /__w/1/s/src/mono/mono/metadata/loader.c:1044
#8  mono_get_method_checked (image=0xab1a6b090b80, token=721422599, klass=<optimized out>, context=0xab1a6b1a7198, error=0xff066892f968) at /__w/1/s/src/mono/mono/metadata/loader.c:1199
#9  0x0000ff069454e780 in mini_get_method_allow_open (m=0xab1a6b2eac80, token=<optimized out>, klass=0x0, context=0xab1a6b1a7198, error=0xff066892f968) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:5093
#10 mono_method_to_ir (cfg=<optimized out>, method=<optimized out>, start_bblock=0xff0668979620, end_bblock=0xff0668979760, return_var=<optimized out>, inline_args=<optimized out>, inline_offset=<optimized out>, is_virtual_call=<optimized out>) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:7684
#11 0x0000ff0694534b5c in inline_method (cfg=0xff066892f460, cmethod=<optimized out>, fsig=<optimized out>, sp=0xff0668978570, ip=0xff06920fc8fb "(\f\t", real_offset=<optimized out>, inline_always=<optimized out>, is_empty=<optimized out>) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:4834
#12 0x0000ff069454e730 in mono_method_to_ir (cfg=<optimized out>, method=<optimized out>, start_bblock=0xff0668977750, end_bblock=0xff0668977890, return_var=<optimized out>, inline_args=<optimized out>, inline_offset=<optimized out>, is_virtual_call=<optimized out>) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:7903
#13 0x0000ff0694534b5c in inline_method (cfg=0xff066892f460, cmethod=<optimized out>, fsig=<optimized out>, sp=0xff0668963e48, ip=0xff06920f65fb "(\004\021", real_offset=<optimized out>, inline_always=<optimized out>, is_empty=<optimized out>) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:4834
#14 0x0000ff069454e730 in mono_method_to_ir (cfg=<optimized out>, method=<optimized out>, start_bblock=0xff0668963380, end_bblock=0xff06689634c0, return_var=<optimized out>, inline_args=<optimized out>, inline_offset=<optimized out>, is_virtual_call=<optimized out>) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:7903
#15 0x0000ff0694534b5c in inline_method (cfg=0xff066892f460, cmethod=<optimized out>, fsig=<optimized out>, sp=0xff06689514a8, ip=0xff06920498e3 "(V\003", real_offset=<optimized out>, inline_always=<optimized out>, is_empty=<optimized out>) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:4834
#16 0x0000ff069454e730 in mono_method_to_ir (cfg=<optimized out>, method=<optimized out>, start_bblock=0xff0668950158, end_bblock=0xff0668950298, return_var=<optimized out>, inline_args=<optimized out>, inline_offset=<optimized out>, is_virtual_call=<optimized out>) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:7903
#17 0x0000ff0694534b5c in inline_method (cfg=0xff066892f460, cmethod=<optimized out>, fsig=<optimized out>, sp=0xff066894ff70, ip=0xff0692049c53 "(\032\021", real_offset=<optimized out>, inline_always=<optimized out>, is_empty=<optimized out>) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:4834
#18 0x0000ff069454e730 in mono_method_to_ir (cfg=<optimized out>, method=<optimized out>, start_bblock=0xff066894f8a0, end_bblock=0xff066894f9e0, return_var=<optimized out>, inline_args=<optimized out>, inline_offset=<optimized out>, is_virtual_call=<optimized out>) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:7903
#19 0x0000ff0694534b5c in inline_method (cfg=0xff066892f460, cmethod=<optimized out>, fsig=<optimized out>, sp=0xff0668929578, ip=0xff0689c115e0 "(\204\006", real_offset=<optimized out>, inline_always=<optimized out>, is_empty=<optimized out>) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:4834
#20 0x0000ff069454e730 in mono_method_to_ir (cfg=<optimized out>, method=<optimized out>, start_bblock=0xff0668021e68, end_bblock=0xff0668021fa8, return_var=<optimized out>, inline_args=<optimized out>, inline_offset=<optimized out>, is_virtual_call=<optimized out>) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:7903
#21 0x0000ff0694525458 in mini_method_compile (method=<optimized out>, opts=374417919, flags=JIT_FLAG_RUN_CCTORS, parts=0, aot_method_index=-1) at /__w/1/s/src/mono/mono/mini/mini.c:3492
#22 0x0000ff0694527e5c in mono_jit_compile_method_inner (method=0xff06682b8e28, opt=-1998317088, error=0xff0688e44110) at /__w/1/s/src/mono/mono/mini/mini.c:4126
#23 0x0000ff069452cb80 in mono_jit_compile_method_with_opt (method=0xff06682b8e28, opt=374417919, jit_only=0, error=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2718
#24 jit_compile_method_with_opt_cb (arg=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2773
#25 jit_compile_method_with_opt (params=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2789
#26 0x0000ff069452bfac in mono_jit_compile_method (method=<optimized out>, error=0xff0688e44110) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2808
#27 0x0000ff06945c83c0 in common_call_trampoline (regs=0xff0688e441a0, code=0xff0667decfd4 "\240", <incomplete sequence \373>, m=0xff06682b8e28, vt=0x0, vtable_slot=<optimized out>, error=0xff0688e44110) at /__w/1/s/src/mono/mono/mini/mini-trampolines.c:618
#28 0x0000ff06945c7ee0 in mono_magic_trampoline (regs=0xff0688e441a0, code=0xff0667decfd4 "\240", <incomplete sequence \373>, arg=0xff06682b8e28, tramp=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-trampolines.c:759
#29 0x0000ff069416b304 in ?? ()
#30 0x0000ff0688e44450 in ?? ()
#31 0x0000ff0693c8d6a8 in ?? ()
Backtrace stopped: not enough registers or memory available to unwind further

Thread 9 (Thread 0xff068906a1b0 (LWP 32) ".NET TP Worker"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0xff0689069320, clockid=<optimized out>, expected=0, futex_word=0xff0689069398) at ../sysdeps/nptl/futex-internal.h:323
#1  __pthread_cond_wait_common (abstime=0xff0689069320, clockid=<optimized out>, mutex=0xff0678036f40, cond=0xff0689069370) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0xff0689069370, mutex=0xff0678036f40, abstime=0xff0689069320) at pthread_cond_wait.c:656
#3  0x0000ff06946b7a9c in mono_os_cond_timedwait (cond=0xff0689069370, mutex=0xff0678036f40, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/mono-os-mutex.c:75
#4  0x0000ff06946bcbbc in mono_coop_cond_timedwait (cond=0xff0689069370, mutex=<optimized out>, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/../../mono/utils/mono-coop-mutex.h:103
#5  mono_lifo_semaphore_timed_wait (semaphore=0xff0678036f40, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/lifo-semaphore.c:48
#6  0x0000ff0689079c38 in ?? ()
#7  0x0000ff0693d9ae80 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 8 (Thread 0xff06890cb1b0 (LWP 31) ".NET TP Gate"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0xff06890ca2a8, clockid=<optimized out>, expected=0, futex_word=0xff0670005488) at ../sysdeps/nptl/futex-internal.h:323
#1  __pthread_cond_wait_common (abstime=0xff06890ca2a8, clockid=<optimized out>, mutex=0xff0670005430, cond=0xff0670005460) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0xff0670005460, mutex=0xff0670005430, abstime=0xff06890ca2a8) at pthread_cond_wait.c:656
#3  0x0000ff069404dc88 in SystemNative_LowLevelMonitor_TimedWait (monitor=0xff0670005430, timeoutMilliseconds=500) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195
#4  0x0000ff0689080744 in ?? ()
#5  0x0000ff06895cef08 in ?? ()
#6  0x0000ff06890ca8e0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 7 (Thread 0xff06895c71b0 (LWP 30) ".NET TP Worker"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0xff06895c60e8, clockid=<optimized out>, expected=0, futex_word=0xff067c005348) at ../sysdeps/nptl/futex-internal.h:323
#1  __pthread_cond_wait_common (abstime=0xff06895c60e8, clockid=<optimized out>, mutex=0xff067c0052f0, cond=0xff067c005320) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0xff067c005320, mutex=0xff067c0052f0, abstime=0xff06895c60e8) at pthread_cond_wait.c:656
#3  0x0000ff069404dc88 in SystemNative_LowLevelMonitor_TimedWait (monitor=0xff067c0052f0, timeoutMilliseconds=12000) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195
#4  0x0000ff0689080744 in ?? ()
#5  0x0000ff06895cef08 in ?? ()
#6  0x0000ff06895c68e0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 6 (Thread 0xff06897d81b0 (LWP 29) ".NET Long Runni"):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0xff06780057e8) at ../sysdeps/nptl/futex-internal.h:186
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xff0678005790, cond=0xff06780057c0) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0xff06780057c0, mutex=0xff0678005790) at pthread_cond_wait.c:638
#3  0x0000ff069404db24 in SystemNative_LowLevelMonitor_Wait (monitor=0xff0678005790) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x0000ff06895cf4cc in ?? ()
#5  0x0000ff0693d0b850 in ?? ()
#6  0x0000ff0693d0c8f0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 5 (Thread 0xff06898ff1b0 (LWP 26) ".NET SigHandler"):
#0  __libc_read (nbytes=1, buf=0xff06898fe977, fd=<optimized out>) at ../sysdeps/unix/sysv/linux/read.c:26
#1  __libc_read (fd=<optimized out>, buf=0xff06898fe977, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:24
#2  0x0000ff069404d1d8 in SignalHandlerLoop (arg=0xab1a6b6b20c0) at /__w/1/s/src/native/libs/System.Native/pal_signal.c:331
#3  0x0000ff0694d9f648 in start_thread (arg=0xff06898feab0) at pthread_create.c:477
#4  0x0000ff0694a3ac1c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 4 (Thread 0xff0691c601b0 (LWP 25) "Finalizer"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0xff0694801950 <finalizer_sem>) at ../sysdeps/nptl/futex-internal.h:323
#1  do_futex_wait (sem=sem@entry=0xff0694801950 <finalizer_sem>, abstime=0x0, clockid=0) at sem_waitcommon.c:112
#2  0x0000ff0694da933c in __new_sem_wait_slow (sem=0xff0694801950 <finalizer_sem>, abstime=0x0, clockid=0) at sem_waitcommon.c:184
#3  0x0000ff0694774514 in mono_os_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/metadata/../utils/mono-os-semaphore.h:204
#4  mono_coop_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/metadata/../../mono/utils/mono-coop-semaphore.h:41
#5  finalizer_thread (unused=<optimized out>) at /__w/1/s/src/mono/mono/metadata/gc.c:891
#6  0x0000ff069474e744 in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1202
#7  0x0000ff069474e5f0 in start_wrapper (data=0xab1a6b14a3f0) at /__w/1/s/src/mono/mono/metadata/threads.c:1264
#8  0x0000ff0694d9f648 in start_thread (arg=0xff0691c5fab0) at pthread_create.c:477
#9  0x0000ff0694a3ac1c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 3 (Thread 0xff0691e611b0 (LWP 24) ".NET EventPipe"):
#0  0x0000ff0694a30ef4 in __GI___poll (fds=0xff06840034f0, nfds=1, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
#1  0x0000ff0694699474 in ipc_poll_fds (fds=<optimized out>, nfds=1, timeout=4294967295) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:470
#2  ds_ipc_poll (poll_handles_data=0xff06840032e0, poll_handles_data_len=1, timeout_ms=4294967295, callback=0xff0694698850 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:1098
#3  0x0000ff0694696968 in ds_ipc_stream_factory_get_next_available_stream (callback=0xff0694698850 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc.c:385
#4  0x0000ff0694695004 in server_thread (data=<optimized out>) at /__w/1/s/src/native/eventpipe/ds-server.c:129
#5  0x0000ff0694698830 in ep_rt_thread_mono_start_func (data=0xab1a6b125300) at /__w/1/s/src/mono/mono/mini/../eventpipe/ep-rt-mono.h:880
#6  0x0000ff0694d9f648 in start_thread (arg=0xff0691e60ab0) at pthread_create.c:477
#7  0x0000ff0694a3ac1c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 2 (Thread 0xff0693bff1b0 (LWP 23) "SGen worker"):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0xff069480fe80 <work_cond+40>) at ../sysdeps/nptl/futex-internal.h:186
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xff069480fe28 <lock>, cond=0xff069480fe58 <work_cond>) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0xff069480fe58 <work_cond>, mutex=0xff069480fe28 <lock>) at pthread_cond_wait.c:638
#3  0x0000ff06947d1e20 in mono_os_cond_wait (cond=0xff069480fe80 <work_cond+40>, mutex=<optimized out>) at /__w/1/s/src/mono/mono/sgen/../../mono/utils/mono-os-mutex.h:219
#4  get_work (worker_index=<optimized out>, work_context=<optimized out>, do_idle=<optimized out>, job=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:167
#5  thread_func (data=0x0) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:198
#6  0x0000ff0694d9f648 in start_thread (arg=0xff0693bfeab0) at pthread_create.c:477
#7  0x0000ff0694a3ac1c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 1 (Thread 0xff0694dec010 (LWP 22) "dotnet"):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0xab1a6c0e25ac) at ../sysdeps/nptl/futex-internal.h:186
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xab1a6c0e2550, cond=0xab1a6c0e2580) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0xab1a6c0e2580, mutex=0xab1a6c0e2550) at pthread_cond_wait.c:638
#3  0x0000ff069404db24 in SystemNative_LowLevelMonitor_Wait (monitor=0xab1a6c0e2550) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x0000ff06895cf4cc in ?? ()
#5  0x0000ff0693d071b0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
[Inferior 1 (process 22) detached]

=================================================================
	Basic Fault Address Reporting
=================================================================
instruction pointer is NULL, skip dumping
=================================================================
	Managed Stacktrace:
=================================================================
	  at <unknown> <0xffffffff>
	  at System.Reflection.RuntimeMethodInfo:InternalInvoke <0x00007>
	  at System.Reflection.MethodInvoker:InterpretedInvoke <0x00073>
	  at System.Reflection.MethodInvoker:Invoke <0x00103>
	  at System.Reflection.RuntimeMethodInfo:Invoke <0x001a7>
	  at System.Reflection.MethodBase:Invoke <0x00053>
	  at Xunit.Sdk.TestInvoker`1:CallTestMethod <0x00047>
	  at <<InvokeTestMethodAsync>b__1>d:MoveNext <0x00377>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0009f>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x0002f>
	  at <>c__DisplayClass48_0:<InvokeTestMethodAsync>b__1 <0x000cb>
	  at <AggregateAsync>d__4:MoveNext <0x000c3>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0009b>
	  at Xunit.Sdk.ExecutionTimer:AggregateAsync <0x000eb>
	  at <>c__DisplayClass48_0:<InvokeTestMethodAsync>b__0 <0x0017b>
	  at <RunAsync>d__9:MoveNext <0x00067>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x00093>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x000e7>
	  at <InvokeTestMethodAsync>d__48:MoveNext <0x00287>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000a7>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002f>
	  at Xunit.Sdk.TestInvoker`1:InvokeTestMethodAsync <0x00113>
	  at Xunit.Sdk.XunitTestInvoker:InvokeTestMethodAsync <0x0011b>
	  at <<RunAsync>b__47_0>d:MoveNext <0x00673>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000af>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002f>
	  at Xunit.Sdk.TestInvoker`1:<RunAsync>b__47_0 <0x000d3>
	  at <RunAsync>d__10`1:MoveNext <0x00073>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x00093>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x000e7>
	  at Xunit.Sdk.TestInvoker`1:RunAsync <0x0010b>
	  at Xunit.Sdk.XunitTestRunner:InvokeTestMethodAsync <0x000cb>
	  at <InvokeTestAsync>d__4:MoveNext <0x001c3>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000a3>
	  at Xunit.Sdk.XunitTestRunner:InvokeTestAsync <0x000ef>
	  at <>c__DisplayClass43_0:<RunAsync>b__0 <0x00043>
	  at <RunAsync>d__10`1:MoveNext <0x00083>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0009f>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002f>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x00113>
	  at <RunAsync>d__43:MoveNext <0x0048f>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000af>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002f>
	  at Xunit.Sdk.TestRunner`1:RunAsync <0x000d3>
	  at Xunit.Sdk.XunitTestCaseRunner:RunTestAsync <0x000df>
	  at <RunAsync>d__19:MoveNext <0x0031f>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000a7>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002f>
	  at Xunit.Sdk.TestCaseRunner`1:RunAsync <0x000cf>
	  at Xunit.Sdk.XunitTestCase:RunAsync <0x000b3>
	  at Xunit.Sdk.XunitTestMethodRunner:RunTestCaseAsync <0x0009f>
	  at <RunTestCasesAsync>d__32:MoveNext <0x001c3>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000af>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002f>
	  at Xunit.Sdk.TestMethodRunner`1:RunTestCasesAsync <0x000d3>
	  at <RunAsync>d__31:MoveNext <0x0019f>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0009f>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002f>
	  at Xunit.Sdk.TestMethodRunner`1:RunAsync <0x000cb>
	  at Xunit.Sdk.XunitTestClassRunner:RunTestMethodAsync <0x000df>
	  at <RunTestMethodsAsync>d__38:MoveNext <0x007b3>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000b7>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002f>
	  at Xunit.Sdk.TestClassRunner`1:RunTestMethodsAsync <0x000d7>
	  at <RunAsync>d__37:MoveNext <0x0033f>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000a7>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002f>
	  at Xunit.Sdk.TestClassRunner`1:RunAsync <0x000cf>
	  at Xunit.Sdk.XunitTestCollectionRunner:RunTestClassAsync <0x000e3>
	  at <RunTestClassesAsync>d__28:MoveNext <0x003cf>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000af>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002f>
	  at Xunit.Sdk.TestCollectionRunner`1:RunTestClassesAsync <0x000d3>
	  at <RunAsync>d__27:MoveNext <0x0033f>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000a7>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002f>
	  at Xunit.Sdk.TestCollectionRunner`1:RunAsync <0x000cf>
	  at Xunit.Sdk.XunitTestAssemblyRunner:RunTestCollectionAsync <0x000af>
	  at <>c__DisplayClass14_2:<RunTestCollectionsAsync>b__2 <0x0006f>
	  at System.Threading.Tasks.Task`1:InnerInvoke <0x0006f>
	  at <>c:<.cctor>b__273_0 <0x0003b>
	  at System.Threading.ExecutionContext:RunInternal <0x000bf>
	  at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x00257>
	  at System.Threading.Tasks.Task:ExecuteEntry <0x000bb>
	  at <>c:<.cctor>b__8_0 <0x00067>
	  at Xunit.Sdk.MaxConcurrencySyncContext:RunOnSyncContext <0x0004f>
	  at <>c__DisplayClass11_0:<WorkerThreadProc>b__0 <0x00053>
	  at System.Threading.ExecutionContext:RunInternal <0x000bf>
	  at System.Threading.ExecutionContext:Run <0x00047>
	  at System.Object:lambda_method2 <0x0008b>
	  at Xunit.Sdk.ExecutionContextHelper:Run <0x00063>
	  at Xunit.Sdk.MaxConcurrencySyncContext:WorkerThreadProc <0x0022f>
	  at <>c:<QueueUserWorkItem>b__5_0 <0x00077>
	  at System.Threading.Tasks.Task:InnerInvoke <0x000b7>
	  at <>c:<.cctor>b__273_0 <0x0003b>
	  at System.Threading.ExecutionContext:RunInternal <0x000bf>
	  at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x00257>
	  at System.Threading.Tasks.Task:ExecuteEntryUnsafe <0x000af>
	  at <>c:<.cctor>b__10_0 <0x0006b>
	  at System.Threading.Thread:StartCallback <0x0012b>
	  at System.Object:runtime_invoke_void__this__ <0x00087>
=================================================================

steveisok · 2023-03-16T00:30:56Z

This is a dupe of #81123

dotnet-issue-labeler bot added the area-VM-meta-mono label Jan 27, 2023

ghost added the untriaged New issue has not been triaged by the area owner label Jan 27, 2023

danmoseley mentioned this issue Jan 27, 2023

Improve error for missing Python #81202

Merged

lambdageek self-assigned this Feb 7, 2023

SamMonoRT removed the untriaged New issue has not been triaged by the area owner label Feb 7, 2023

SamMonoRT added this to the 8.0.0 milestone Feb 7, 2023

SamMonoRT mentioned this issue Feb 7, 2023

Roslyn source generator crash on mono/linux/arm64 #81123

Closed

steveharter mentioned this issue Feb 16, 2023

Add reflection introspection support for function pointers #81006

Merged

danmoseley changed the title ~~Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests segfault in monoeg_g_hash_table_insert_replace~~ Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests segfault in Mono Feb 16, 2023

bartonjs mentioned this issue Feb 16, 2023

Hold temporary keychain refs for certificates imported with X509Certificate2Collection.Import #82205

Merged

carlossanlop mentioned this issue Mar 16, 2023

Remove System.Drawing.Common from runtime #83356

Merged

steveisok closed this as completed Mar 16, 2023

ghost locked as resolved and limited conversation to collaborators Apr 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests segfault in Mono #81249

Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests segfault in Mono #81249

danmoseley commented Jan 27, 2023 •

edited by lambdageek

Loading

stephentoub commented Jan 27, 2023

danmoseley commented Feb 7, 2023

janvorli commented Feb 7, 2023

danmoseley commented Feb 7, 2023

SamMonoRT commented Feb 7, 2023

lambdageek commented Feb 7, 2023 •

edited

Loading

steveisok commented Feb 15, 2023

lewing commented Feb 15, 2023

lambdageek commented Feb 16, 2023

lambdageek commented Feb 17, 2023 •

edited

Loading

carlossanlop commented Mar 15, 2023

steveisok commented Mar 16, 2023

Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests segfault in Mono #81249

Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests segfault in Mono #81249

Comments

danmoseley commented Jan 27, 2023 • edited by lambdageek Loading

stephentoub commented Jan 27, 2023

danmoseley commented Feb 7, 2023

janvorli commented Feb 7, 2023

danmoseley commented Feb 7, 2023

SamMonoRT commented Feb 7, 2023

lambdageek commented Feb 7, 2023 • edited Loading

steveisok commented Feb 15, 2023

lewing commented Feb 15, 2023

lambdageek commented Feb 16, 2023

lambdageek commented Feb 17, 2023 • edited Loading

carlossanlop commented Mar 15, 2023

steveisok commented Mar 16, 2023

danmoseley commented Jan 27, 2023 •

edited by lambdageek

Loading

lambdageek commented Feb 7, 2023 •

edited

Loading

lambdageek commented Feb 17, 2023 •

edited

Loading