Disabled Job - Libraries Test Run release mono windows x64 Release #45524

riarenas · 2020-12-03T02:46:44Z

Almost all tests involving networking fail in that leg. The test runs have been disabled temporarily in #45529

An example in https://dnceng.visualstudio.com/public/_build/results?buildId=906071&view=ms.vss-test-web.build-test-results-tab

[6:10 PM] Alexander Köplinger
hmm this might be related to the helix rollout since it overlaps with when the rollout happened. all of the workitems have something to do with networking or hitting some http, and some have exceptions when trying to instantiante MsQuicApi
it's possible that a new Windows version got rolled out that enables http3/quic somehow and that triggers this codepath

Exploratory PR in #45520

Dotnet-GitSync-Bot · 2020-12-03T02:46:46Z

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

scalablecory · 2020-12-03T10:00:20Z

Looking at logs, failures appear to be related to reflection. Can you point me to where you are seeing MsQuicApi as possible suspect?

some have exceptions when trying to instantiante MsQuicApi

A first-chance exception is expected here as a P/Invoke call is used to test for a DLL's presence. Are you seeing the exception leak out in other ways?

it's possible that a new Windows version got rolled out that enables http3/quic somehow and that triggers this codepath

Even with an updated Windows, unless the new version of Windows includes the required msquic.dll to light up the .NET feature -- I have been told this is explicitly not the plan for MsQuic -- I would not expect any change in behavior. CC @nibanks

akoeplinger · 2020-12-03T11:38:45Z

One example is in this log:

System.Xml.Tests.TC_SchemaSet_Add_URL.v4 [FAIL]
      System.NullReferenceException : Object reference not set to an instance of an object
      Stack Trace:
        /_/src/mono/netcore/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.cs(378,0): at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

Unhandled Exception:
System.TypeInitializationException: The type initializer for 'System.Net.Quic.Implementations.MsQuic.Internal.MsQuicApi' threw an exception.

 ---> System.NullReferenceException: Object reference not set to an instance of an object

   at System.Net.Quic.Implementations.MsQuic.Internal.MsQuicApi..ctor() in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Implementations/MsQuic/Internal/MsQuicApi.cs:line 28

   at System.Net.Quic.Implementations.MsQuic.Internal.MsQuicApi..cctor() in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Implementations/MsQuic/Internal/MsQuicApi.cs:line 156

   --- End of inner exception stack trace ---
[ERROR] FATAL UNHANDLED EXCEPTION: System.TypeInitializationException: The type initializer for 'System.Net.Quic.Implementations.MsQuic.Internal.MsQuicApi' threw an exception.

 ---> System.NullReferenceException: Object reference not set to an instance of an object

   at System.Net.Quic.Implementations.MsQuic.Internal.MsQuicApi..ctor() in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Implementations/MsQuic/Internal/MsQuicApi.cs:line 28

   at System.Net.Quic.Implementations.MsQuic.Internal.MsQuicApi..cctor() in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Implementations/MsQuic/Internal/MsQuicApi.cs:line 156

   --- End of inner exception stack trace ---

It sounds like something goes wrong when loading the MsQuic library but I don't see any related changes between the good/bad commits so my current hunch is that something in the OS changed.

akoeplinger · 2020-12-03T12:37:36Z

Hmm after more digging this doesn't look related to MsQuic anymore, I tried disabling loading it with #45520 but there are still other workitems that hit the reflection issue.

We seem to be hitting it on the release/5.0 branch as well so it definitely looks related to the recent Helix rollout.

riarenas · 2020-12-03T15:38:49Z

@dotnet/dnceng for visibility. This looks like it started failing on all windows queues as of the latest rollout.

premun · 2020-12-03T15:46:12Z

I opened https://github.com/dotnet/core-eng/issues/11569 for tracking

eerhardt · 2020-12-03T16:43:27Z

I hit the same NullReferenceException in https://dev.azure.com/dnceng/public/_build/results?buildId=906896&view=logs&j=d81dbc02-ec0b-5bd5-14c1-0072bd710d3b&t=780f1b30-8338-5e4d-6b72-c57eeb2413c9&l=76

Executed on a000GUM

C:\h\w\A14F08F6\w\AB370972\e>call RunTests.cmd --runtime-path C:\h\w\A14F08F6\p 
----- start Thu 12/03/2020  4:22:57.90 ===============  To repro directly: ===================================================== 
pushd C:\h\w\A14F08F6\w\AB370972\e\
"C:\h\w\A14F08F6\p\dotnet.exe" exec --runtimeconfig System.Data.Common.Tests.runtimeconfig.json --depsfile System.Data.Common.Tests.deps.json xunit.console.dll System.Data.Common.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================

C:\h\w\A14F08F6\w\AB370972\e>"C:\h\w\A14F08F6\p\dotnet.exe" exec --runtimeconfig System.Data.Common.Tests.runtimeconfig.json --depsfile System.Data.Common.Tests.deps.json xunit.console.dll System.Data.Common.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Data.Common.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Data.Common.Tests (found 1810 of 1813 test cases)
  Starting:    System.Data.Common.Tests (parallel test collections = on, max threads = 2)
    System.Data.Tests.SqlTypes.SqlStringTest.ReadWriteXmlTest [FAIL]
      System.NullReferenceException : Object reference not set to an instance of an object
      Stack Trace:
        /_/src/mono/netcore/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.cs(378,0): at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
    System.Data.Common.Tests.DbConnectionTests.CanBeFinalized [SKIP]
      Condition(s) not met: "IsPreciseGcSupported"
    System.Data.Common.Tests.DbCommandTests.CanBeFinalized [SKIP]
      Condition(s) not met: "IsPreciseGcSupported"
   System.Data.Common.Tests: [Long Running Test] 'System.Data.Tests.SqlTypes.SqlInt64Test.ReadWriteXmlTest', Elapsed: 00:02:11
   System.Data.Common.Tests: [Long Running Test] 'System.Data.Tests.SqlTypes.SqlInt64Test.ReadWriteXmlTest', Elapsed: 00:04:11
   System.Data.Common.Tests: [Long Running Test] 'System.Data.Tests.SqlTypes.SqlInt64Test.ReadWriteXmlTest', Elapsed: 00:06:11
   System.Data.Common.Tests: [Long Running Test] 'System.Data.Tests.SqlTypes.SqlInt64Test.ReadWriteXmlTest', Elapsed: 00:08:11
   System.Data.Common.Tests: [Long Running Test] 'System.Data.Tests.SqlTypes.SqlInt64Test.ReadWriteXmlTest', Elapsed: 00:10:11
   System.Data.Common.Tests: [Long Running Test] 'System.Data.Tests.SqlTypes.SqlInt64Test.ReadWriteXmlTest', Elapsed: 00:12:11
   System.Data.Common.Tests: [Long Running Test] 'System.Data.Tests.SqlTypes.SqlInt64Test.ReadWriteXmlTest', Elapsed: 00:14:11

ghost · 2020-12-03T16:49:06Z

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

Almost all tests involving networking fail in that leg. The test runs have been disabled temporarily in #45529

An example in https://dnceng.visualstudio.com/public/_build/results?buildId=906071&view=ms.vss-test-web.build-test-results-tab

[6:10 PM] Alexander Köplinger
hmm this might be related to the helix rollout since it overlaps with when the rollout happened. all of the workitems have something to do with networking or hitting some http, and some have exceptions when trying to instantiante MsQuicApi
it's possible that a new Windows version got rolled out that enables http3/quic somehow and that triggers this codepath

Exploratory PR in #45520

Author:	riarenas
Assignees:	-
Labels:	`area-Infrastructure-mono`, `blocking-clean-ci`, `untriaged`
Milestone:	-

eerhardt · 2020-12-03T16:51:45Z

Also hit in another PR #45522.

Log: https://dev.azure.com/dnceng/public/_build/results?buildId=906830&view=logs&j=d81dbc02-ec0b-5bd5-14c1-0072bd710d3b&t=780f1b30-8338-5e4d-6b72-c57eeb2413c9&l=76

MattGal · 2020-12-03T17:00:28Z

Investigating via https://github.com/dotnet/core-eng/issues/11569

akoeplinger · 2020-12-03T17:02:37Z

@eerhardt those builds ran before the leg was disabled.

akoeplinger · 2020-12-03T17:07:48Z

I think this might be related to the VS update since I can reproduce it locally, i.e. the cause is not the Helix test machine but
the build machine.

akoeplinger · 2020-12-03T18:30:28Z

Ok, I confirmed that this is indeed due to the VS update.

Good:

  **********************************************************************
  ** Visual Studio 2019 Developer Command Prompt v16.7.3
  ** Copyright (c) 2020 Microsoft Corporation
  **********************************************************************
  [vcvarsall.bat] Environment initialized for: 'x86_x64'
  -- Building for: Visual Studio 16 2019
  -- The C compiler identification is MSVC 19.27.29111.0
  -- The CXX compiler identification is MSVC 19.27.29111.0

Bad:

  **********************************************************************
  ** Visual Studio 2019 Developer Command Prompt v16.8.0
  ** Copyright (c) 2020 Microsoft Corporation
  **********************************************************************
  [vcvarsall.bat] Environment initialized for: 'x86_x64'
  -- Building for: Visual Studio 16 2019
  -- The C compiler identification is MSVC 19.28.29333.0
  -- The CXX compiler identification is MSVC 19.28.29333.0

akoeplinger · 2020-12-03T18:33:42Z

To reproduce:

.\build.cmd mono+libs+libs.pretest -c Release

.\dotnet.cmd build /p:RuntimeFlavor=mono /t:Test /p:Configuration=Release .\src\libraries\System.Private.Xml\tests\XmlSchema\XmlSchemaSet\System.Xml.XmlSchemaSet.Tests.csproj

radical · 2020-12-03T23:18:05Z

BatchedCI build for 2ee13ec is also failing with NRE in RuntimeMethodInfo.Invoke.

After upgrade to later msvc version on CI boots, see dotnet#45524 for details, Window x64 Release builds started to crash on libraries tests. After investigation it turns out that new msvc compiler handles an expression different compared to how it was handled in previous version. After upgrade of msvc, the expression: int _amd64_width_temp = ((guint64)(imm) == (guint64)(int)(guint64)(imm)); implemented in amd64_mov_reg_imm and then called from tramp-amd64.c@500 was transformed into an always true expression by compiler: amd64_mov_reg_imm (code, AMD64_R11, (guint8*)mono_get_rethrow_preserve_exception_addr ()); lea rcx,[rethrow_preserve_exception_func (07FFB9E33A590h)] mov word ptr [rbx+0Dh],0BB41h mov byte ptr [rbx+0Fh],cl mov rax,rcx shr eax,8 mov byte ptr [rbx+10h],al mov rax,rcx shr eax,10h shr ecx,18h mov byte ptr [rbx+11h],al lea rax,[rbx+13h] mov byte ptr [rbx+12h],cl as seen above, the condition and handling of a 64-bit imm has been dropped by compiler. This cause issues when the imm is a 64-bit value since it will always gets truncated into 32-bit imm and in this case it was a pointer to a function within coreclr.dll (mono_get_rethrow_preserve_exception_addr) loaded located at higher address (using more than 32-bit). This is most likely a regression issue in compiler for this specific construction. I tried simpler construction (using same type conversion) on both old and new compiler version and then it makes the right optimization. Fix is to switch to a macro already available in amd64-codegen (amd64_is_imm32) detecting if an imm needs a 32-bit or 64-bit sized value. This will be correctly optimized by new msvc compiler and even if this is a work around for a what seems to be a optimization bug in the compiler, it is still cleaner and better describes the intent than current code. Fix also re-enable Windows x64 Release CI test lane.

lateralusX · 2021-01-05T10:44:46Z

Crash should be fixed by #46573. PR also re-enables the Windows x64 Release CI lane.

After upgrade to later msvc version on CI boots, see dotnet/runtime#45524 for details, Window x64 Release builds started to crash on libraries tests. After investigation it turns out that new msvc compiler handles an expression different compared to how it was handled in previous version. After upgrade of msvc, the expression: int _amd64_width_temp = ((guint64)(imm) == (guint64)(int)(guint64)(imm)); implemented in amd64_mov_reg_imm and then called from tramp-amd64.c@500 was transformed into an always true expression by compiler: ``` amd64_mov_reg_imm (code, AMD64_R11, (guint8*)mono_get_rethrow_preserve_exception_addr ()); lea rcx,[rethrow_preserve_exception_func (07FFB9E33A590h)] mov word ptr [rbx+0Dh],0BB41h mov byte ptr [rbx+0Fh],cl mov rax,rcx shr eax,8 mov byte ptr [rbx+10h],al mov rax,rcx shr eax,10h shr ecx,18h mov byte ptr [rbx+11h],al lea rax,[rbx+13h] mov byte ptr [rbx+12h],cl ``` as seen above, the condition and handling of a 64-bit imm has been dropped by compiler. This cause issues when the imm is a 64-bit value since it will always gets truncated into 32-bit imm and in this case it was a pointer to a function within coreclr.dll (mono_get_rethrow_preserve_exception_addr) loaded located at higher address (using more than 32-bit). This is most likely a regression issue in compiler for this specific construction. I tried simpler construction (using same type conversion) on both old and new compiler version and then it makes the right optimization. Fix is to switch to a macro already available in amd64-codegen (amd64_is_imm32) detecting if an imm needs a 32-bit or 64-bit sized value. This will be correctly optimized by new msvc compiler and even if this is a work around for a what seems to be a optimization bug in the compiler, it is still cleaner and better describes the intent than current code. Fix also re-enable Windows x64 Release CI test lane.

danmoseley · 2021-01-10T18:56:26Z

Do we need to report to compiler team?

lateralusX · 2021-01-11T07:42:24Z

@danmosemsft Yes that would be good, how do we best progress regarding that?

After upgrade to later msvc version on CI boots, see dotnet/runtime#45524 for details, Window x64 Release builds started to crash on libraries tests. After investigation it turns out that new msvc compiler handles an expression different compared to how it was handled in previous version. After upgrade of msvc, the expression: int _amd64_width_temp = ((guint64)(imm) == (guint64)(int)(guint64)(imm)); implemented in amd64_mov_reg_imm and then called from tramp-amd64.c@500 was transformed into an always true expression by compiler: ``` amd64_mov_reg_imm (code, AMD64_R11, (guint8*)mono_get_rethrow_preserve_exception_addr ()); lea rcx,[rethrow_preserve_exception_func (07FFB9E33A590h)] mov word ptr [rbx+0Dh],0BB41h mov byte ptr [rbx+0Fh],cl mov rax,rcx shr eax,8 mov byte ptr [rbx+10h],al mov rax,rcx shr eax,10h shr ecx,18h mov byte ptr [rbx+11h],al lea rax,[rbx+13h] mov byte ptr [rbx+12h],cl ``` as seen above, the condition and handling of a 64-bit imm has been dropped by compiler. This cause issues when the imm is a 64-bit value since it will always gets truncated into 32-bit imm and in this case it was a pointer to a function within coreclr.dll (mono_get_rethrow_preserve_exception_addr) loaded located at higher address (using more than 32-bit). This is most likely a regression issue in compiler for this specific construction. I tried simpler construction (using same type conversion) on both old and new compiler version and then it makes the right optimization. Fix is to switch to a macro already available in amd64-codegen (amd64_is_imm32) detecting if an imm needs a 32-bit or 64-bit sized value. This will be correctly optimized by new msvc compiler and even if this is a work around for a what seems to be a optimization bug in the compiler, it is still cleaner and better describes the intent than current code. Fix also re-enable Windows x64 Release CI test lane. Co-authored-by: lateralusX <lateralusX@users.noreply.github.com>

After upgrade to later msvc version on CI boots, see #45524 for details, Window x64 Release builds started to crash on libraries tests. After investigation it turns out that new msvc compiler handles an expression different compared to how it was handled in previous version. After upgrade of msvc, the expression: int _amd64_width_temp = ((guint64)(imm) == (guint64)(int)(guint64)(imm)); implemented in amd64_mov_reg_imm and then called from tramp-amd64.c@500 was transformed into an always true expression by compiler: amd64_mov_reg_imm (code, AMD64_R11, (guint8*)mono_get_rethrow_preserve_exception_addr ()); lea rcx,[rethrow_preserve_exception_func (07FFB9E33A590h)] mov word ptr [rbx+0Dh],0BB41h mov byte ptr [rbx+0Fh],cl mov rax,rcx shr eax,8 mov byte ptr [rbx+10h],al mov rax,rcx shr eax,10h shr ecx,18h mov byte ptr [rbx+11h],al lea rax,[rbx+13h] mov byte ptr [rbx+12h],cl as seen above, the condition and handling of a 64-bit imm has been dropped by compiler. This cause issues when the imm is a 64-bit value since it will always gets truncated into 32-bit imm and in this case it was a pointer to a function within coreclr.dll (mono_get_rethrow_preserve_exception_addr) loaded located at higher address (using more than 32-bit). This is most likely a regression issue in compiler for this specific construction. I tried simpler construction (using same type conversion) on both old and new compiler version and then it makes the right optimization. Fix is to switch to a macro already available in amd64-codegen (amd64_is_imm32) detecting if an imm needs a 32-bit or 64-bit sized value. This will be correctly optimized by new msvc compiler and even if this is a work around for a what seems to be a optimization bug in the compiler, it is still cleaner and better describes the intent than current code. Fix also re-enable Windows x64 Release CI test lane.

akoeplinger · 2021-01-11T14:58:42Z

@lateralusX I'd start with https://docs.microsoft.com/en-us/cpp/overview/how-to-report-a-problem-with-the-visual-cpp-toolset which essentially means opening an issue on developer community via https://aka.ms/feedback/report?space=62.

If we don't get a response we can ping someone internally.

akoeplinger · 2021-01-11T15:00:49Z

Closing the issue since the problem on our end was fixed with #46573

riarenas added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Dec 3, 2020

Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label Dec 3, 2020

hoyosjs changed the title ~~multiple tests in Libraries Test Run release mono windows x64 Release failing~~ Disabled Job - Libraries Test Run release mono windows x64 Release Dec 3, 2020

safern mentioned this issue Dec 3, 2020

DisableFileSystemWatcherTest #45511

Merged

akoeplinger mentioned this issue Dec 3, 2020

[runtime] Avoid checking for hardened runtime on ios, its not needed, and it causes alerts because of PROT_WRITE|PROT_EXEC. #45514

Merged

runfoapp bot mentioned this issue Dec 3, 2020

Infrastructure - Status/Health #702

Closed

eerhardt mentioned this issue Dec 3, 2020

Remove unnecessary IsStatic checks in ActivatorUtilities #45439

Merged

jeffschwMSFT added the area-Infrastructure-mono label Dec 3, 2020

eerhardt mentioned this issue Dec 3, 2020

Ensure XslCompiledTransform works with TrimMode=link #45522

Merged

akoeplinger removed the untriaged New issue has not been triaged by the area owner label Dec 3, 2020

akoeplinger assigned lateralusX Dec 3, 2020

akoeplinger added this to the 6.0.0 milestone Dec 3, 2020

riarenas mentioned this issue Dec 3, 2020

Disable Long Running JSON tests on Checked CoreCLR #45505

Merged

This was referenced Dec 10, 2020

[release/5.0] Fix debugger attach+read on Linux 32 bit #45882

Merged

[release/5.0] Backport #45529: Disable Windows x64 Mono test leg #45887

Merged

akoeplinger mentioned this issue Dec 10, 2020

Temporarily disable Mono Desktop Windows Release x64 #45529

Merged

lateralusX mentioned this issue Jan 5, 2021

Fix Mono Windows x64 crash blocking CI Windows x64 Release job. #46573

Merged

monojenkins mentioned this issue Jan 5, 2021

Fix Mono Windows x64 crash blocking CI Windows x64 Release job. mono/mono#20720

Merged

akoeplinger closed this as completed Jan 11, 2021

ghost locked as resolved and limited conversation to collaborators Feb 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disabled Job - Libraries Test Run release mono windows x64 Release #45524

Disabled Job - Libraries Test Run release mono windows x64 Release #45524

riarenas commented Dec 3, 2020 •

edited by hoyosjs

Loading

Dotnet-GitSync-Bot commented Dec 3, 2020

scalablecory commented Dec 3, 2020 •

edited

Loading

akoeplinger commented Dec 3, 2020 •

edited

Loading

akoeplinger commented Dec 3, 2020

riarenas commented Dec 3, 2020

premun commented Dec 3, 2020

eerhardt commented Dec 3, 2020

ghost commented Dec 3, 2020

eerhardt commented Dec 3, 2020

MattGal commented Dec 3, 2020

akoeplinger commented Dec 3, 2020

akoeplinger commented Dec 3, 2020

akoeplinger commented Dec 3, 2020

akoeplinger commented Dec 3, 2020

radical commented Dec 3, 2020

lateralusX commented Jan 5, 2021

danmoseley commented Jan 10, 2021

lateralusX commented Jan 11, 2021

akoeplinger commented Jan 11, 2021

akoeplinger commented Jan 11, 2021

Disabled Job - Libraries Test Run release mono windows x64 Release #45524

Disabled Job - Libraries Test Run release mono windows x64 Release #45524

Comments

riarenas commented Dec 3, 2020 • edited by hoyosjs Loading

Dotnet-GitSync-Bot commented Dec 3, 2020

scalablecory commented Dec 3, 2020 • edited Loading

akoeplinger commented Dec 3, 2020 • edited Loading

akoeplinger commented Dec 3, 2020

riarenas commented Dec 3, 2020

premun commented Dec 3, 2020

eerhardt commented Dec 3, 2020

ghost commented Dec 3, 2020

eerhardt commented Dec 3, 2020

MattGal commented Dec 3, 2020

akoeplinger commented Dec 3, 2020

akoeplinger commented Dec 3, 2020

akoeplinger commented Dec 3, 2020

akoeplinger commented Dec 3, 2020

radical commented Dec 3, 2020

lateralusX commented Jan 5, 2021

danmoseley commented Jan 10, 2021

lateralusX commented Jan 11, 2021

akoeplinger commented Jan 11, 2021

akoeplinger commented Jan 11, 2021

riarenas commented Dec 3, 2020 •

edited by hoyosjs

Loading

scalablecory commented Dec 3, 2020 •

edited

Loading

akoeplinger commented Dec 3, 2020 •

edited

Loading