Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Hot Reload] metadata.c:1326, condition `GINT_TO_UINT32(idx) < table_info_get_rows (t)' not met #93860

Closed
rolfbjarne opened this issue Oct 23, 2023 · 50 comments
Assignees
Labels
arch-wasm WebAssembly architecture area-EnC-mono Hot Reload for WebAssembly, iOS/Android, etc os-ios Apple iOS
Milestone

Comments

@rolfbjarne
Copy link
Member

From @jeromelaban on Fri, 20 Oct 2023 03:53:31 GMT

Steps to Reproduce

  1. Create a maui app (net8 rc2)
  2. Add a [CreateNewOnMetadataUpdate] attribute on MainPage
  3. Build the app
  4. Run the app
  5. Add a ToString(); call in the ctor, or any other MainPage modification
  6. Perform a C# hot reload

Expected Behavior

The code is updated without crashing.

Actual Behavior

Sometimes:

error: * Assertion at /Users/runner/work/1/s/src/mono/mono/metadata/metadata.c:1326, condition `GINT_TO_UINT32(idx) < table_info_get_rows (t)' not met

Sometimes:

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x104aba1d5 - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmonosgen-2.0.dylib : mono_dump_native_crash_info
	0x104a58b6e - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmonosgen-2.0.dylib : mono_handle_native_crash
	0x1049abaff - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmonosgen-2.0.dylib : mono_sigsegv_signal_handler_debug
	0x7ff8377405ed - /usr/lib/system/libsystem_platform.dylib : _sigtramp
	0x0 - Unknown
	0x103b33fd6 - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmono-component-debugger.dylib : get_types_for_source_file
	0x103b3ce1d - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmono-component-debugger.dylib : monoeg_g_hash_table_foreach
	0x103b25917 - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmono-component-debugger.dylib : mono_process_dbg_packet
	0x103b2d3ff - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmono-component-debugger.dylib : mono_debugger_agent_receive_and_process_command
	0x103b300f0 - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmono-component-debugger.dylib : debugger_thread
	0x104bbb4fe - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmonosgen-2.0.dylib : start_wrapper
	0x7ff83774d1d3 - /usr/lib/system/libsystem_pthread.dylib : _pthread_start
	0x7ff837748bd3 - /usr/lib/system/libsystem_pthread.dylib : thread_start

Environment

  • VS 2022 17.8 Preview 4
  • .NET 8.0.100-rc.2.23502.2
Installed Workload Id      Manifest Version                     Installation Source
----------------------------------------------------------------------------------------------------------------------
maui-ios                   8.0.0-rc.2.9373/8.0.100-rc.2         VS 17.7.34009.444
ios                        16.4.8968-net8-rc2/8.0.100-rc.2      VS 17.7.34009.444, VS 17.8.34212.112, VS 17.8.34112.27
maccatalyst                16.4.8968-net8-rc2/8.0.100-rc.2      VS 17.7.34009.444, VS 17.8.34212.112, VS 17.8.34112.27
maui-android               8.0.0-rc.2.9373/8.0.100-rc.2         VS 17.7.34009.444
android                    34.0.0-rc.2.468/8.0.100-rc.2         VS 17.7.34009.444, VS 17.8.34212.112, VS 17.8.34112.27
maui-windows               8.0.0-rc.2.9373/8.0.100-rc.2         VS 17.7.34009.444, VS 17.8.34212.112, VS 17.8.34112.27
maui-maccatalyst           8.0.0-rc.2.9373/8.0.100-rc.2         VS 17.7.34009.444

Build Logs

Available on demand.

Copied from original issue xamarin/xamarin-macios#19277

@rolfbjarne
Copy link
Member Author

From @rolfbjarne on Fri, 20 Oct 2023 08:44:49 GMT

@jeromelaban this looks like a crash in the debugger; does the app run if you don't attach a debugger (just tap on it)?

@rolfbjarne
Copy link
Member Author

From @jeromelaban on Fri, 20 Oct 2023 11:30:28 GMT

@rolfbjarne thanks for looking into it! No, the app does not crash, but there's no hot reload that can happen either. The issue only happens when explicitly triggering a hot reload (which happens through the debugger in this case).

It may be a runtime issue, though, since it fails in a very similar way under Android.

I can open a runtime issue if you think it's more appropriate.

@rolfbjarne
Copy link
Member Author

From @jeromelaban on Fri, 20 Oct 2023 12:36:24 GMT

@rolfbjarne I just discovered that I missed some steps in the original description, my apologies. Hot reloading is required for this to fail.

@rolfbjarne
Copy link
Member Author

From @rolfbjarne on Mon, 23 Oct 2023 12:15:07 GMT

I can open a runtime issue if you think it's more appropriate.

Yes, this looks like a runtime issue, so the dotnet/runtime repository would be a better place for it.

I'll move it there.

@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Oct 23, 2023
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Oct 23, 2023
@jeromelaban
Copy link
Contributor

jeromelaban commented Oct 23, 2023

@rolfbjarne I was under the impression that using System.Reflection.Metadata.MetadataUpdater.ApplyUpdate was more stable, but it's still failing with this same error (when compared to the debugger doing the same through its own APIs), sometimes immediately, sometimes after a few updates.

@jeromelaban
Copy link
Contributor

Also, the issue happens on webassembly, which makes sense as it's mono underneath. (/cc @lewing)

@lambdageek
Copy link
Member

CreateNewOnMetadataUpdate is hard to use (the old class name is still around and valid) and I don't think Maui has been updated to use it (although I could be wrong). But the runtime crash is not expected.

@lambdageek lambdageek added area-EnC-mono Hot Reload for WebAssembly, iOS/Android, etc and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Oct 23, 2023
@lambdageek
Copy link
Member

@jeromelaban do you have separate repro steps for Wasm? I think it's probably crashing in a different place for the same underlying reason. It would be helpful to see the other stack trace

@jeromelaban
Copy link
Contributor

@lambdageek I'll try to build one using blazor.

In the meantime it really is almost the same place, probably in slight version differences:

[MONO] * Assertion at /__w/runtime/src/mono/mono/metadata/metadata.c:1321, condition `GINT_TO_UINT32(idx) < table_info_get_rows (t)' not met

    at mono_wasm_stringify_as_error_with_stack (dotnet.js:5:620)
    at Object.mono_wasm_trace_logger (dotnet.js:5:985)
    at _mono_wasm_trace_logger (dotnet.js:14:111110)
    at dotnet.wasm:0x9f18
    at dotnet.wasm:0xabc71
    at dotnet.wasm:0x1c8ae2
    at dotnet.wasm:0x1c8b41
    at dotnet.wasm:0x1c8b75
    at dotnet.wasm:0x822de
    at dotnet.wasm:0x821ff

@jeromelaban
Copy link
Contributor

jeromelaban commented Oct 24, 2023

I'm still not able to get a repro with blazor, but here's another log, which includes the call to dump_update_summary:
93860-crash_trace.txt

Here's an excerpt of the log:

[MONO] no implementation for interface method testhr01.MainPage#1.IMainPage_Bindings::Initialize() in class testhr01.MainPage#1/MainPage_Bindings
[MONO] METHOD get_Owner()
[MONO] METHOD set_Owner(testhr01.MainPage#1)
[MONO] METHOD .ctor(testhr01.MainPage#1)
[MONO] METHOD testhr01.MainPage.IMainPage_Bindings.Initialize()
[MONO] METHOD testhr01.MainPage.IMainPage_Bindings.Update()
[MONO] METHOD testhr01.MainPage.IMainPage_Bindings.UpdateResources()
[MONO] METHOD testhr01.MainPage.IMainPage_Bindings.StopTracking()
[MONO] Running class .cctor for System.SR from 'System.Private.CoreLib.dll'
[MONO] Running class .cctor for System.Resources.ResourceManager from 'System.Private.CoreLib.dll'
[MONO] Running class .cctor for System.Resources.ResourceReader from 'System.Private.CoreLib.dll'
[MONO] Running class .cctor for System.Resources.FastResourceComparer from 'System.Private.CoreLib.dll'
Error: [MONO] * Assertion at /__w/Uno.DotnetRuntime.WebAssembly/Uno.DotnetRuntime.WebAssembly/runtime/src/mono/mono/metadata/metadata.c:1321, condition `GINT_TO_UINT32(idx) < table_info_get_rows (t)' not met

Update: Still trying to get a repro, and it seems to be related to nested types marked as CreateNewOnMetadataUpdate. Will update as I find more.

@lambdageek
Copy link
Member

@jeromelaban unfortunately this part ofhte trace:

[MONO] * Assertion at /__w/runtime/src/mono/mono/metadata/metadata.c:1321, condition `GINT_TO_UINT32(idx) < table_info_get_rows (t)' not met

    at mono_wasm_stringify_as_error_with_stack (dotnet.js:5:620)
    at Object.mono_wasm_trace_logger (dotnet.js:5:985)
    at _mono_wasm_trace_logger (dotnet.js:14:111110)

is not useful. that's just the part where the runtime is reporting that there was a failed assertion.

what I really need is the symbolic names for this part:

 at dotnet.wasm:0x9f18
    at dotnet.wasm:0xabc71
    at dotnet.wasm:0x1c8ae2
    at dotnet.wasm:0x1c8b41
    at dotnet.wasm:0x1c8b75
    at dotnet.wasm:0x822de
    at dotnet.wasm:0x821ff

The line in metadata.c:1321 is in mono_metadata_decode_row_raw which is called from from around 200 places.

On ios it is clearly something in the debugger. But on wasm I wouldn't expect the updates to come in via the debugger (and I don't think the debugger is active right - you're just running dotnet-watch?) so it's some other place.

@lewing @pavelsavara do we have some way so symbolicate wasm crashes?

@lambdageek lambdageek added this to the 8.0.x milestone Oct 24, 2023
@lambdageek lambdageek added arch-wasm WebAssembly architecture os-ios Apple iOS and removed untriaged New issue has not been triaged by the area owner labels Oct 24, 2023
@ghost
Copy link

ghost commented Oct 24, 2023

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

From @jeromelaban on Fri, 20 Oct 2023 03:53:31 GMT

Steps to Reproduce

  1. Create a maui app (net8 rc2)
  2. Add a [CreateNewOnMetadataUpdate] attribute on MainPage
  3. Build the app
  4. Run the app
  5. Add a ToString(); call in the ctor, or any other MainPage modification
  6. Perform a C# hot reload

Expected Behavior

The code is updated without crashing.

Actual Behavior

Sometimes:

error: * Assertion at /Users/runner/work/1/s/src/mono/mono/metadata/metadata.c:1326, condition `GINT_TO_UINT32(idx) < table_info_get_rows (t)' not met

Sometimes:

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x104aba1d5 - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmonosgen-2.0.dylib : mono_dump_native_crash_info
	0x104a58b6e - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmonosgen-2.0.dylib : mono_handle_native_crash
	0x1049abaff - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmonosgen-2.0.dylib : mono_sigsegv_signal_handler_debug
	0x7ff8377405ed - /usr/lib/system/libsystem_platform.dylib : _sigtramp
	0x0 - Unknown
	0x103b33fd6 - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmono-component-debugger.dylib : get_types_for_source_file
	0x103b3ce1d - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmono-component-debugger.dylib : monoeg_g_hash_table_foreach
	0x103b25917 - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmono-component-debugger.dylib : mono_process_dbg_packet
	0x103b2d3ff - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmono-component-debugger.dylib : mono_debugger_agent_receive_and_process_command
	0x103b300f0 - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmono-component-debugger.dylib : debugger_thread
	0x104bbb4fe - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmonosgen-2.0.dylib : start_wrapper
	0x7ff83774d1d3 - /usr/lib/system/libsystem_pthread.dylib : _pthread_start
	0x7ff837748bd3 - /usr/lib/system/libsystem_pthread.dylib : thread_start

Environment

  • VS 2022 17.8 Preview 4
  • .NET 8.0.100-rc.2.23502.2
Installed Workload Id      Manifest Version                     Installation Source
----------------------------------------------------------------------------------------------------------------------
maui-ios                   8.0.0-rc.2.9373/8.0.100-rc.2         VS 17.7.34009.444
ios                        16.4.8968-net8-rc2/8.0.100-rc.2      VS 17.7.34009.444, VS 17.8.34212.112, VS 17.8.34112.27
maccatalyst                16.4.8968-net8-rc2/8.0.100-rc.2      VS 17.7.34009.444, VS 17.8.34212.112, VS 17.8.34112.27
maui-android               8.0.0-rc.2.9373/8.0.100-rc.2         VS 17.7.34009.444
android                    34.0.0-rc.2.468/8.0.100-rc.2         VS 17.7.34009.444, VS 17.8.34212.112, VS 17.8.34112.27
maui-windows               8.0.0-rc.2.9373/8.0.100-rc.2         VS 17.7.34009.444, VS 17.8.34212.112, VS 17.8.34112.27
maui-maccatalyst           8.0.0-rc.2.9373/8.0.100-rc.2         VS 17.7.34009.444

Build Logs

Available on demand.

Copied from original issue xamarin/xamarin-macios#19277

Author: rolfbjarne
Assignees: lambdageek
Labels:

arch-wasm, os-ios, area-EnC-mono

Milestone: 8.0.x

@lambdageek
Copy link
Member

I was under the impression that using System.Reflection.Metadata.MetadataUpdater.ApplyUpdate was more stable

This is not the case, btw. The debugger and the managed API call down into identical code. The only way in which not using the debugger is more stable is that there is less code observing the hot reload changes. (The debugger does things like query the names of locals, get all the members of a class, etc etc, that typical code could do with reflection, but is pretty rare in general).

@jeromelaban
Copy link
Contributor

Thanks for the updates @lambdageek.

is not useful. that's just the part where the runtime is reporting that there was a failed assertion.

Agreed it's not useful at this point, I'm trying to get a repro for blazor specifically and have symbols for that trace as well, no luck so far (having both the crash and symbols enabled).

But on wasm I wouldn't expect the updates to come in via the debugger (and I don't think the debugger is active right - you're just running dotnet-watch?) so it's some other place.

For wasm it's not using the debugger, indeed, the browserlink feature is used to provide the updates.

Aside from that, on the iOS front from my own Uno tests, I enable traces for everything, and here's what I got:
93860-crash_trace_ios.txt

2023-10-24 09:40:15.521066-0400 testhr04.Mobile[94697:35244945] info: >>> EnC delta for base=/Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/9BA8BC6D-7CA1-4E02-9D18-107DE14D9B72/testhr04.Mobile.app/testhr04.dll (generation 2) applied
2023-10-24 09:40:15.705544-0400 testhr04.Mobile[94697:35244905] info: NO Found interfaces for class 0x0200000f
2023-10-24 09:40:15.705696-0400 testhr04.Mobile[94697:35244905] info: Creating class '.IMainPage_Bindings' from skeleton (first_method_idx = 0x00000053, count = 0x00000004, first_field_idx = 0x00000000, count=0x00000000)
2023-10-24 09:40:15.705888-0400 testhr04.Mobile[94697:35244905] debug: Requesting loading reference 18 (of 25) of /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/9BA8BC6D-7CA1-4E02-9D18-107DE14D9B72/testhr04.Mobile.app/testhr04.dll
2023-10-24 09:40:15.706043-0400 testhr04.Mobile[94697:35244905] debug: Loading reference 18 of /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/9BA8BC6D-7CA1-4E02-9D18-107DE14D9B72/testhr04.Mobile.app/testhr04.dll (default ALC), looking for System.Runtime, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
2023-10-24 09:40:15.706245-0400 testhr04.Mobile[94697:35244905] debug: Request to load System.Runtime in alc 0x600003695300
2023-10-24 09:40:15.706378-0400 testhr04.Mobile[94697:35244905] debug: Assembly already loaded in the active ALC: 'System.Runtime'.
2023-10-24 09:40:15.706493-0400 testhr04.Mobile[94697:35244905] debug: Assembly Ref addref testhr04[0x60000389ce10] -> System.Runtime[0x600003884000]: 107
2023-10-24 09:40:15.706624-0400 testhr04.Mobile[94697:35244905] info: Found interfaces for class 0x02000010 starting at 0x00000003
2023-10-24 09:40:15.706757-0400 testhr04.Mobile[94697:35244905] info: Creating class '.MainPage_Bindings' from skeleton (first_method_idx = 0x00000057, count = 0x00000007, first_field_idx = 0x00000032, count=0x00000001)
2023-10-24 09:40:15.706860-0400 testhr04.Mobile[94697:35244905] info: NO Found interfaces for class 0x02000011
2023-10-24 09:40:15.706984-0400 testhr04.Mobile[94697:35244905] info: Creating class '.<>c' from skeleton (first_method_idx = 0x0000005e, count = 0x00000007, first_field_idx = 0x00000033, count=0x00000006)
2023-10-24 09:40:15.708790-0400 testhr04.Mobile[94697:35244905] debug: Requesting loading reference 19 (of 25) of /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/9BA8BC6D-7CA1-4E02-9D18-107DE14D9B72/testhr04.Mobile.app/testhr04.dll
2023-10-24 09:40:15.708885-0400 testhr04.Mobile[94697:35244905] debug: Loading reference 19 of /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/9BA8BC6D-7CA1-4E02-9D18-107DE14D9B72/testhr04.Mobile.app/testhr04.dll (default ALC), looking for System.Runtime.Loader, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
2023-10-24 09:40:15.709068-0400 testhr04.Mobile[94697:35244905] debug: Request to load System.Runtime.Loader in alc 0x600003695300
2023-10-24 09:40:15.709230-0400 testhr04.Mobile[94697:35244905] debug: Assembly already loaded in the active ALC: 'System.Runtime.Loader'.
2023-10-24 09:40:15.709352-0400 testhr04.Mobile[94697:35244905] debug: Assembly Ref addref testhr04[0x60000389ce10] -> System.Runtime.Loader[0x60000389c1b0]: 8
2023-10-24 09:40:15.709485-0400 testhr04.Mobile[94697:35244905] debug: Requesting loading reference 20 (of 25) of /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/9BA8BC6D-7CA1-4E02-9D18-107DE14D9B72/testhr04.Mobile.app/testhr04.dll
2023-10-24 09:40:15.709620-0400 testhr04.Mobile[94697:35244905] debug: Loading reference 20 of /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/9BA8BC6D-7CA1-4E02-9D18-107DE14D9B72/testhr04.Mobile.app/testhr04.dll (default ALC), looking for Microsoft.iOS, Version=16.4.0.0, Culture=neutral, PublicKeyToken=84e04ff9cfb79065
2023-10-24 09:40:15.709741-0400 testhr04.Mobile[94697:35244905] debug: Request to load Microsoft.iOS in alc 0x600003695300
2023-10-24 09:40:15.709862-0400 testhr04.Mobile[94697:35244905] debug: Assembly already loaded in the active ALC: 'Microsoft.iOS'.
2023-10-24 09:40:15.709957-0400 testhr04.Mobile[94697:35244905] debug: Assembly Ref addref testhr04[0x60000389ce10] -> Microsoft.iOS[0x60000389c120]: 17
2023-10-24 09:40:16.318353-0400 testhr04.Mobile[94697:35244905] info: NO Found interfaces for class 0x0200000e
2023-10-24 09:40:16.318669-0400 testhr04.Mobile[94697:35244905] debug: method lookup idx=0x0000004c returned gen=2 il=0x7fc15cdbed2c
2023-10-24 09:40:16.318839-0400 testhr04.Mobile[94697:35244905] debug: method lookup idx=0x0000004c returned gen=2 il=0x600004828530
2023-10-24 09:40:16.319023-0400 testhr04.Mobile[94697:35244905] debug: member_parent lookup: 0x0400002f returned 0x0200000e
2023-10-24 09:40:16.319194-0400 testhr04.Mobile[94697:35244905] debug: member_parent lookup: 0x04000030 returned 0x0200000e
2023-10-24 09:40:16.333531-0400 testhr04.Mobile[94697:35244905] error: * Assertion at /Users/runner/work/1/s/src/mono/mono/metadata/metadata.c:1326, condition `GINT_TO_UINT32(idx) < table_info_get_rows (t)' not met

That being said, the crash on MAUI is somewhat different (using the repro steps at the top): 93860-crash_trace_ios_maui.txt, so maybe it's two separate issues, after all.

@jeromelaban
Copy link
Contributor

jeromelaban commented Oct 25, 2023

@lambdageek I've been trying to get to a repro for the metadata.c issue inside a MAUI app, but I can't get the same conditions that fail. Here's a repro that uses Uno Platform and vscode on mac to repro the crash for iOS:

To repro on macOS:

  • Install the uno vscode addin, it should be version 0.11.0 or later (to have launch without debugger working)
  • Extract the repro: issue93860.zip
  • Open the folder in vscode
  • Wait omnisharp to load (You may need to disable C# devkit temporarily and enable the useOmnisharp setting)
  • In the bottom bar, instead of issue93860.sln, select issue93860.Mobile.csproj
  • Wait for the platforms list to be populated and select net8.0-ios | Debug and the appropriate simulator
  • In "Debug" vertical left tab, select Uno Platform Mobile for the debug target
  • Press Ctrl+F5 or Run Without Debugging (important because HR cannot happen with the debugger attached with Uno)
  • Once the app is started, open the issue93860/MainPage.xaml
  • Wait a few seconds for the "Uno Platform - Hot Reload" output tab to finish loading
  • Change Hello Uno Platform to something else then save the file, multiple times (The "Uno Platform - Hot Reload" output should show "Found 1 metadata updates after ..."

Notice that the app will crash with the metadata.c:1326 assertion.

The app has all runtime debugging enabled for easier troubleshooting.

@lambdageek
Copy link
Member

@jeromelaban I can repro the Uno crash. It's always on the second update for me.

crash reporter shows this stack:

8   libsystem_c.dylib             	       0x1241ecd60 abort + 133
9   libxamarin-dotnet-debug.dylib 	       0x10ae01340 log_callback(char const*, char const*, char const*, int, void*) + 64
10  libmonosgen-2.0.dylib         	       0x10b508632 monoeg_g_logv_nofree + 194
11  libmonosgen-2.0.dylib         	       0x10b5087af monoeg_assertion_message + 143
12  libmonosgen-2.0.dylib         	       0x10b5087ea mono_assertion_message + 26
13  libmonosgen-2.0.dylib         	       0x10b58d364 mono_metadata_decode_row + 436
14  libmonosgen-2.0.dylib         	       0x10b552801 mono_ppdb_lookup_location_internal + 65
15  libmonosgen-2.0.dylib         	       0x10b59c6d8 mono_debug_lookup_source_location + 312
16  libmonosgen-2.0.dylib         	       0x10b4548ca mono_get_trace + 698
17  libmonosgen-2.0.dylib         	       0x10b566a8b ves_icall_System_Diagnostics_StackTrace_GetTrace + 43
18  libmonosgen-2.0.dylib         	       0x10b4cdde8 do_icall + 248
19  libmonosgen-2.0.dylib         	       0x10b4cc60e do_icall_wrapper + 350
20  libmonosgen-2.0.dylib         	       0x10b4bd2f6 mono_interp_exec_method + 3990
21  libmonosgen-2.0.dylib         	       0x10b4ce1e5 interp_entry + 453
22  libmonosgen-2.0.dylib         	       0x10b4cea17 interp_entry_static_1 + 55

which looks like there was already some fault and then the StackTrace.GetTrace method was invoked and it crashed in the ppdb infrastructure.

I'm going to try to make some custom builds of the runtime for iossimulator with extra logging.

by the way, is it possible to build the project for iossimulator-arm64. My naive attempts failed (the build tried to link some SkiaSharp native library that was built for ios not iossimulator and at that point I gave up). Not a big deal (I can make iossimulator-x64 custom runtimes, so I'm not blocked), just curious.

@jeromelaban
Copy link
Contributor

@lambdageek it is supposed to be supported, that's what SkiaSharp 2.88.6 was about, but I'll try on arm64 and let you know if there's something missing (could be the native assets for iOS that are incorrect). We could also completely remove all skiasharp references, they're not needed for this issue, but I'll post an updated sample without it so you don't have to fiddle.

@jeromelaban
Copy link
Contributor

jeromelaban commented Oct 25, 2023

@lambdageek I was not able to fail the build with the original sample on arm64, but here's an updated one without Skia dependencies: issue83860-2.zip.

@jeromelaban
Copy link
Contributor

@lambdageek Furthermore, seeing the stacktrace that you posted got me thinking that the interpreter could play a role, but it's not, even when disabling it.

Adding:

<MtouchExtraArgs>$(MtouchExtraArgs) --setenv=DOTNET_MODIFIABLE_ASSEMBLIES=debug</MtouchExtraArgs>

(because there's no interpreter) does not help and the failure in metadata.c is the same.

@lambdageek
Copy link
Member

If the PR above is good enough I can validate it on Wasm, if that can help!

yea that would be useful. although i'm not sure if wasm has the same problems or different problems :D

I'm still trying to understand why in the Uno iossimulator repro something is trying to throw an exception after a hot reload change is applied. The table_info_get_rows crash is masking some other problem - and i'm trying to figure out what that is.

@fanyang-mono
Copy link
Member

@jeromelaban with the new zip file, when I following your reproduction steps, I ran into the following error.

/Users/yangfan/Documents/work/issue83860/issue93860/bin/Debug/net8.0-ios/iossimulator-x64/issue93860.app could not be installed on simulator F584177B-85D7-4B39-8627-7EB09B8A69C0.
An error was encountered processing the command (domain=NSPOSIXErrorDomain, code=2):
Simulator device returned an error for the requested operation.
An application bundle was not found at the provided path.
Provide a valid path to the desired application bundle.
Underlying error (domain=NSPOSIXErrorDomain, code=2):
	Failed to install the requested application
	An application bundle was not found at the provided path.

It seems that issue93860.app was not generated. I couldn't find it anywhere. Do you know what I was missing?

@jeromelaban
Copy link
Contributor

@fanyang-mono thanks for trying out the repro! This is curious that you're not getting the app to build. In the terminal that builds the app, are there any errors that are showing up? Otherwise, there's a binlog that is generated in the folder of the mobile project which may contain more information about what's not working.

I'm also wondering, it seems that you're building for x64, are you on an x64 mac as well?

@fanyang-mono
Copy link
Member

@jeromelaban Yes, I am using mac x64

@jeromelaban
Copy link
Contributor

@fanyang-mono thanks, then the build error may be specific to your local setup for iOS and iOS workloads, somehow. The binlog I mentioned is very likely to provide additional information.

@jeromelaban
Copy link
Contributor

jeromelaban commented Nov 9, 2023

@lambdageek I just finished validating the fix from #94540 on Wasm, it works! Thanks for working on it!

@lambdageek
Copy link
Member

lambdageek commented Nov 9, 2023

@jeromelaban I'm getting some weird behavior when I try to use my local build of Microsoft.NETCore.App.Runtime.Mono.iossimulator-x64 to build the repro app. (What I do is I go into $DOTNET_ROOT/packs/Microsoft.NETCore.App.Runtime.Mono.iossimulator-x64/8.0.0-rc.2.23479.6 and overwrite all the files with the contents of my locally-built nuget. it's crude, but it (usually?) works).

I end up with a crash like this:

14  libmonosgen-2.0.dylib                    0x10f360d12 mono_jit_thread_attach + 178
15  issue93860.Mobile                        0x104d62e65 native_to_managed_trampoline_1(objc_object*, objc_selector*, _MonoMethod**, bool*, unsigned int) + 133 (registrar.mm:15)
16  issue93860.Mobile                        0x104d65a02 -[issue93860_AppHead init] + 50 (registrar.mm:13726)
17  UIKitCore                                0x1340ac44f _UIApplicationMainPreparations + 1585
18  UIKitCore                                0x1340abdf4 UIApplicationMain + 95
19  libxamarin-dotnet-debug.dylib            0x10ed884da xamarin_UIApplicationMain + 58
20  ???                                      0x164305b92 ???
21  libmonosgen-2.0.dylib                    0x129342499 mono_jit_runtime_invoke + 1769
22  libmonosgen-2.0.dylib                    0x1295215e7 mono_runtime_invoke_checked + 135
23  libmonosgen-2.0.dylib                    0x129528625 mono_runtime_exec_main_checked + 117
24  libmonosgen-2.0.dylib                    0x12939f506 mono_jit_exec + 358
25  libxamarin-dotnet-debug.dylib            0x10edccbba xamarin_main + 1898
26  issue93860.Mobile                        0x104d62cc4 main + 52 (main.x86_64.mm:80)

I added some extra printfs and I can tell that I somehow ended up with two copies of mono_get_root_domain. I don't understand how that can be happening (and seemingly only when I drop my own build on top). Does Uno do anything special to the ios build?

Also I still don't understand what is forcing my VS Code to build for iossimulator-x64 rather than iossimulator-arm64. Not sure if it's related. (When I hit ctrl-F5, the build terminal says: Executing task in folder issue83860: dotnet build "/Users/alklig/work/bugs/gh-93860/ver2/issue83860/issue93860.Mobile/issue93860.Mobile.csproj" -f:net8.0-ios -c:Debug -r:iossimulator-x64 -clp:NoSummary -p:GenerateFullPaths=true -p:UnoForceSingleTFM=true -bl:"/Users/alklig/work/bugs/gh-93860/ver2/issue83860/issue93860.Mobile/net8.0-ios-Debug-iossimulator-x64.binlog" ). So it's something about the Uno extension, I guess?

@lambdageek
Copy link
Member

Turning off the static registrar didn't help. So that's not it.

@jeromelaban
Copy link
Contributor

@lambdageek I'm not sure why x64 is selected, I'm looking into it. But you should be able to build and run directly using dotnet build -f net8.0-ios -r iossimulator-arm64 -t:run (without the debugger though).

@jeromelaban
Copy link
Contributor

Also, you should still be able to test hotreload, if the folder was opened in VS Code.

@lambdageek
Copy link
Member

I think the "overwrite libmonosgen-2.0.dylib" trick doesn't quite work for ios. having trouble with it with a plain (arm64) maui app, too. so I'm going to backport the assertion fix to net8 since it is clearly beneficial (our new unit test shows it helps with new code that throws) but I can't guarantee it will actually fix all the problems for Uno's hot reload on ios.

@rolfbjarne is there some way to validate a runtime change with a released version of the ios SDK? everything I'm trying results in suspicious crashes that look like multiple copies of the runtime in the final app.

@rolfbjarne
Copy link
Member Author

@lambdageek I would probably:

  1. Build the iOS app and inspect the binlog to figure out the exact libmonosgen-2.0.a file that's linked into the app.
  2. Replace that libmonosgen-2.0.a file with the one I want to use.

I believe @ivanpovazan had a different (and possibly easier) way to do this though.

@ivanpovazan
Copy link
Member

@lambdageek one thing that might help is adding the following target in the project file:

<Target Name="UpdateRuntimePackAndAOTCompiler" 
	AfterTargets="ResolveFrameworkReferences"
	Condition="'$(UseCurrentMainBranch)' == 'true'" >
	<Error Condition="'$(DotnetRuntimeRepo)' == ''" Text="You must specify the location of your dotnet/runtime repository via -p:DotnetRuntimeRepo=&lt;some_repo_dir&gt;"/>
	<ItemGroup>
		<!-- update runtime pack to local build -->
		<ResolvedRuntimePack PackageDirectory="$(DotnetRuntimeRepo)/artifacts/bin/microsoft.netcore.app.runtime.ios-arm64/Release"
			Condition="'%(ResolvedRuntimePack.FrameworkName)' == 'Microsoft.NETCore.App'" />
		<!-- remove AOT compiler reference for ios-arm64 -->
		<MonoAotCrossCompiler Remove="@(MonoAotCrossCompiler)"
			Condition="%(RuntimeIdentifier) == 'ios-arm64'" />
		<!-- update AOT compiler reference to local build -->
		<MonoAotCrossCompiler Include="$(DotnetRuntimeRepo)/artifacts/bin/mono/ios.arm64.Release/cross/ios-arm64/mono-aot-cross">
			<RuntimeIdentifier>ios-arm64</RuntimeIdentifier>
		</MonoAotCrossCompiler>
	</ItemGroup>
</Target>

and then build the app by passing in -p:UseCurrentMainBranch=true -p:DotnetRuntimeRepo=<repo_root>
this way you should be able to easily switch between locally built version and the released one.

One thing to note is that the example is configured to target an iOS device.

Hope this helps.

lambdageek added a commit that referenced this issue Nov 13, 2023
…o_debug_lookup_source_location (#94612)

Backport of #94540 to release/8.0-staging

If a newly added method is in a stack trace, look it up in the EnC debug information, if available.

Related to #93860

* Add new crashing test

* [mono] catch the case of updated methods in mono_debug_lookup_source_location

Co-authored-by: Aleksey Kliger <alklig@microsoft.com>
Co-authored-by: Thays Grazia <thaystg@gmail.com>
@lambdageek
Copy link
Member

Hope this helps.

@ivanpovazan that's very helpful. Unfortunately it only works with an actual device, not with a simulator (I adjusted the obvious RID-specific bits of the .csproj snippet). With the simulator I end up getting errors related to the sdb trampoline - which is usually an indication that some JITing is happening.

@jeromelaban Does Uno hot reload work on an actual device? I tried running the app with Ctrl-F5 and making changes, but I didn't see anything show up. The Uno Platform - Hot Reload and Uno Platform - XAML windows both didn't show any additional output after I changed MainPage.xaml and saved.

@lewing
Copy link
Member

lewing commented Nov 13, 2023

@lewing @pavelsavara do we have some way so symbolicate wasm crashes?

The runtime pack should have a dotnet.native.js.symbols to symbolicate @radical can give you more details.

You can also build a Debug runtime locally and update the runtime pack for browser-wasm to use that

@radical
Copy link
Member

radical commented Nov 14, 2023

@lewing @pavelsavara do we have some way so symbolicate wasm crashes?

The runtime pack should have a dotnet.native.js.symbols to symbolicate @radical can give you more details.

We have a symbolicator in src/mono/wasm/symbolicator. You can use it like:

src/mono/wasm/symbolicator$ dotnet run <path-to-dotnet.native.js.symbols-from-the-app> ../data/wasm-symbol-patterns.txt <file-with-trace>

@jeromelaban
Copy link
Contributor

@jeromelaban Does Uno hot reload work on an actual device? I tried running the app with Ctrl-F5 and making changes, but I didn't see anything show up. The Uno Platform - Hot Reload and Uno Platform - XAML windows both didn't show any additional output after I changed MainPage.xaml and saved.

@lambdageek it is supposed to, yes, though because of this issue I wasn't able to test this very far. There may be error messages showing up in the device log, also. You may be able to see more messages when enabling trace logging here:

builder.SetMinimumLevel(LogLevel.Trace);

and here:

builder.AddFilter("Uno.UI.RemoteControl", LogLevel.Trace);

@lambdageek
Copy link
Member

@jeromelaban Could you try .NET SDK 8.0.1

@jeromelaban
Copy link
Contributor

jeromelaban commented Jan 11, 2024

@lambdageek I'd like to, but there are no new available workloads available that I could see. VS 17.8.4 still has the 8.0.100 workloads installed. I tried with https://aka.ms/dotnet/maui/main.json, but those are causing install issues on macOS (not sure of the reason).

Would you know of the workload versions I should be using for 8.0.101?

@lambdageek
Copy link
Member

lambdageek commented Jan 11, 2024

@jeromelaban I don't really understand how workloads work. but with a clean install of 8.0.101 on osx-arm64if I run sudo dotnet workload install maui, I see:

...
Installing pack Microsoft.NETCore.App.Runtime.AOT.osx-arm64.Cross.ios-arm64 version 8.0.1...
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.AOT.Cross.ios-arm64 version 8.0.1...
...
Installing pack Microsoft.NETCore.App.Runtime.Mono.ios-arm64 version 8.0.1...
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.Mono.ios-arm64 version 8.0.1...
...

which I think means it's getting the servicing versions of the runtime packs

I have no idea how to discover this info after the workload installation is completed.

@steveisok tells me the right place to look is sdk-root/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.1/WorkloadManifest.json

@lambdageek
Copy link
Member

This is what I see:

% dotnet workload update --print-rollback
==workloadRollbackDefinitionJsonOutputStart==
{
  "microsoft.net.sdk.android": "34.0.52/8.0.100",
  "microsoft.net.sdk.ios": "17.0.8490/8.0.100",
  "microsoft.net.sdk.maccatalyst": "17.0.8490/8.0.100",
  "microsoft.net.sdk.macos": "14.0.8490/8.0.100",
  "microsoft.net.sdk.maui": "8.0.3/8.0.100",
  "microsoft.net.sdk.tvos": "17.0.8490/8.0.100",
  "microsoft.net.workload.mono.toolchain.current": "8.0.1/8.0.100",
  "microsoft.net.workload.emscripten.current": "8.0.1/8.0.100",
  "microsoft.net.workload.emscripten.net6": "8.0.1/8.0.100",
  "microsoft.net.workload.emscripten.net7": "8.0.1/8.0.100",
  "microsoft.net.workload.mono.toolchain.net6": "8.0.1/8.0.100",
  "microsoft.net.workload.mono.toolchain.net7": "8.0.1/8.0.100",
  "microsoft.net.sdk.aspire": "8.0.0-preview.2.23619.3/8.0.100"
}
==workloadRollbackDefinitionJsonOutputEnd==

maybe that's useful?

@jeromelaban
Copy link
Contributor

@lambdageek yes, it's definitely useful, I did not know about --print-rollback, I'll keep it under my belt :D

Still, those were the versions I was using, and I'm having trouble with validating the upgrade from 8.0.100 to 8.0.101 (on macOS, the workload upgrade does not finish properly). Still that's out of the current issue, so I'll probably open a separate issue for this.

I'll test HR for android/iOS and will report back here! Thanks for following up!

@jeromelaban
Copy link
Contributor

So I've tested on android, and the issue is different now:

After a few reloads (like two or three), I get this:

[libc] Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x8 in tid 17526 (Debugger agent), pid 17504 (nyname.UnoApp16)

Interestingly, there's nothing in ADB that is recognizable, but I may not read it properly. So the current issue may be fixed in 8.0.101 for android but I can't test it...

@jeromelaban
Copy link
Contributor

jeromelaban commented Jan 11, 2024

I'm also getting random crashes when starting the app with this:

[nyname.UnoApp16] * Assertion at /__w/1/s/src/mono/mono/metadata/object.c:657, condition `lock->done' not met

But this is unrelated to hot reload.

@jeromelaban
Copy link
Contributor

So the crash issue is #96872 (threading related), but if I work around it, I can confirm that the repro steps in #93860 (comment) are not failing anymore, thanks!

@lambdageek
Copy link
Member

Ok, the threading thing is because I don't read POSIX manpages closely enough. Closing this issue since the hot reload is working now.

@github-actions github-actions bot locked and limited conversation to collaborators Feb 11, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
arch-wasm WebAssembly architecture area-EnC-mono Hot Reload for WebAssembly, iOS/Android, etc os-ios Apple iOS
Projects
None yet
Development

No branches or pull requests

7 participants