Fatal error reporting deadlocks with GC

[jkotas]

Found while investigating aspnet/KestrelHttpServer#2503

From Maoni:

This is an age old problem…managed heap corruption causes exception triggered on the GC thread which then needs to wait for GC so deadlock…looping in Noah. He did some work to accommodate I believe.

  54  Id: 48c.1b30 Suspend: 0 Teb: 000000ed`15540000 Unfrozen
# Child-SP          RetAddr           Call Site
00 000000ed`27e7c868 00007fff`1be03acf ntdll!NtWaitForSingleObject+0x14
01 000000ed`27e7c870 00007ffe`f8716ba0 KERNELBASE!WaitForSingleObjectEx+0x8f
02 (Inline Function) --------`-------- coreclr!CLREventWaitHelper2+0x6 [e:\a\_work\104\s\src\vm\synch.cpp @ 385] 
03 000000ed`27e7c910 00007ffe`f89de32d coreclr!CLREventWaitHelper+0x20 [e:\a\_work\104\s\src\vm\synch.cpp @ 411] 
04 (Inline Function) --------`-------- coreclr!CLREventBase::Wait+0x11 [e:\a\_work\104\s\src\vm\synch.cpp @ 426] 
05 (Inline Function) --------`-------- coreclr!GCEvent::Impl::Wait+0x11 [e:\a\_work\104\s\src\vm\gcenv.os.cpp @ 789] 
06 (Inline Function) --------`-------- coreclr!GCEvent::Wait+0x18 [e:\a\_work\104\s\src\vm\gcenv.os.cpp @ 867] 
07 000000ed`27e7c970 00007ffe`f86777c3 coreclr!SVR::GCHeap::WaitUntilGCComplete+0x3d [e:\a\_work\104\s\src\gc\gcee.cpp @ 402] 
08 000000ed`27e7c9a0 00007ffe`f869c5d0 coreclr!Thread::RareDisablePreemptiveGC+0x10f [e:\a\_work\104\s\src\vm\threadsuspend.cpp @ 3065] 
09 (Inline Function) --------`-------- coreclr!Thread::DisablePreemptiveGC+0x1f [e:\a\_work\104\s\src\vm\threads.h @ 2107] 
0a (Inline Function) --------`-------- coreclr!GCHolderBase::EnterInternalCoop+0x37 [e:\a\_work\104\s\src\vm\threads.h @ 6068] 
0b 000000ed`27e7c9f0 00007ffe`f889f8a1 coreclr!GCCoop::GCCoop+0x54 [e:\a\_work\104\s\src\vm\threads.h @ 6188] 
0c 000000ed`27e7ca20 00007ffe`f8897e0d coreclr!GetThreadUICultureId+0x51 [e:\a\_work\104\s\src\vm\ceemain.cpp @ 3137] 
0d 000000ed`27e7ca70 00007ffe`f88967e0 coreclr!CCompRC::LoadString+0x55 [e:\a\_work\104\s\src\utilcode\ccomprc.cpp @ 687] 
0e 000000ed`27e7cbb0 00007ffe`f898a19d coreclr!SString::LoadResourceAndReturnHR+0xd8 [e:\a\_work\104\s\src\utilcode\sstring_com.cpp @ 63] 
0f (Inline Function) --------`-------- coreclr!SString::LoadResourceAndReturnHR+0x19 [e:\a\_work\104\s\src\utilcode\sstring_com.cpp @ 27] 
10 (Inline Function) --------`-------- coreclr!SString::LoadResource+0x19 [e:\a\_work\104\s\src\utilcode\sstring_com.cpp @ 21] 
11 000000ed`27e7cc70 00007ffe`f88feb85 coreclr!EventReporter::EventReporter+0x115 [e:\a\_work\104\s\src\vm\eventreporter.cpp @ 56] 
12 000000ed`27e7cf50 00007ffe`f88fe311 coreclr!EEPolicy::LogFatalError+0x15d [e:\a\_work\104\s\src\vm\eepolicy.cpp @ 1264] 
13 000000ed`27e7dcc0 00007ffe`f883cc5b coreclr!EEPolicy::HandleFatalError+0x179 [e:\a\_work\104\s\src\vm\eepolicy.cpp @ 1545] 
14 (Inline Function) --------`-------- coreclr!CLRVectoredExceptionHandlerPhase3+0x12472e [e:\a\_work\104\s\src\vm\excep.cpp @ 7840] 
15 000000ed`27e7e2b0 00007ffe`f87184eb coreclr!CLRVectoredExceptionHandlerPhase2+0x12475f [e:\a\_work\104\s\src\vm\excep.cpp @ 7547] 
16 000000ed`27e7e310 00007ffe`f8718427 coreclr!CLRVectoredExceptionHandler+0x8b [e:\a\_work\104\s\src\vm\excep.cpp @ 7514] 
17 000000ed`27e7e370 00007fff`1f60c810 coreclr!CLRVectoredExceptionHandlerShim+0xb7 [e:\a\_work\104\s\src\vm\excep.cpp @ 8207] 
18 000000ed`27e7e3a0 00007fff`1f5e962b ntdll!RtlInitializeCriticalSection+0x190
19 000000ed`27e7e440 00007fff`1f649c8a ntdll!RtlImageNtHeaderEx+0x17b
1a 000000ed`27e7eb40 00007ffe`f89ea658 ntdll!KiUserExceptionDispatcher+0x3a
1b (Inline Function) --------`-------- coreclr!SVR::CObjectHeader::GetMethodTable+0x7 [e:\a\_work\104\s\src\gc\gc.cpp @ 3962] 
1c (Inline Function) --------`-------- coreclr!SVR::my_get_size+0x7 [e:\a\_work\104\s\src\gc\gc.cpp @ 9007] 
1d 000000ed`27e7f250 00007ffe`f89f7850 coreclr!SVR::gc_heap::find_first_object+0x90 [e:\a\_work\104\s\src\gc\gc.cpp @ 27343] 
1e 000000ed`27e7f2a0 00007ffe`f89f7ec1 coreclr!SVR::gc_heap::revisit_written_page+0x88 [e:\a\_work\104\s\src\gc\gc.cpp @ 26257] 
1f 000000ed`27e7f340 00007ffe`f89e2fd2 coreclr!SVR::gc_heap::revisit_written_pages+0x2c1 [e:\a\_work\104\s\src\gc\gc.cpp @ 26544] 
20 000000ed`27e7f420 00007ffe`f89eb92c coreclr!SVR::gc_heap::background_mark_phase+0x6ca [e:\a\_work\104\s\src\gc\gc.cpp @ 25973] 
21 000000ed`27e7f4c0 00007ffe`f89e5624 coreclr!SVR::gc_heap::gc1+0xc8 [e:\a\_work\104\s\src\gc\gc.cpp @ 15388] 
22 000000ed`27e7f510 00007ffe`f87b98e4 coreclr!SVR::gc_heap::bgc_thread_function+0x120 [e:\a\_work\104\s\src\gc\gc.cpp @ 26991] 
23 000000ed`27e7f560 00007ffe`f8717658 coreclr!<lambda_50311027edc69684c2f6a68020e9a683>::operator()+0x8c [e:\a\_work\104\s\src\vm\gcenv.ee.cpp @ 1238] 
24 000000ed`27e7f590 00007fff`1cd68364 coreclr!Thread::intermediateThreadProc+0x88 [e:\a\_work\104\s\src\vm\threads.cpp @ 2255] 
25 000000ed`27e7fd50 00007fff`1f607091 kernel32!BaseThreadInitThunk+0x14
26 000000ed`27e7fd80 00000000`00000000 ntdll!RtlUserThreadStart+0x21

[jkotas]

This looks like a regression introduced by dotnet/coreclr#16875 and related changes.

[jkotas]

cc @noahfalk @sywhang @danmosemsft

[RussKeldorph]

Fixed by dotnet/coreclr#17844