The active test run was aborted. Reason: Test host process crashed #2261

xuzhg · 2019-12-03T18:44:41Z

Description

For a defect reproducable from the vstest command line, describe the issue you've observed.

Steps to reproduce

git clone https://github.com/xuzhg/WebApi.git
git checkout -b RoutingBaseForNEtCore30 origin/RoutingBaseForNEtCore30
dotnet --version
run the following command (please replace the following Folder):
dotnet test [YOURLocalClonedFolder]\test\E2ETest\Microsoft.Test.E2E.AspNet.OData\Build.AspNetCore3x\Microsoft.Test.E2E.AspNetCore3x.OData.csproj --logger trx --results-directory C:\BuildAgent\_work\_temp --configuration release --no-build -v diag
result:

Expected behavior

Shall pass all test cases running

Actual behavior

Crash with the error message:

    <RunInfos>
      <RunInfo computerName="ODataBuild50" outcome="Error" timestamp="2019-12-02T20:25:22.4166882+00:00">
        <Text>The active test run was aborted. Reason: Test host process crashed : Unhandled exception. Unhandled exception. System.Threading.Tasks.TaskCanceledException: The operation was canceled.&#xD;
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)&#xD;
   at Microsoft.Test.E2E.AspNet.OData.QueryComposition.SecurityTests.TestDosAttack(String filter) in C:\BuildAgent\_work\1\s\test\E2ETest\Microsoft.Test.E2E.AspNet.OData\QueryComposition\SecurityTests.cs:line 85&#xD;
   at Microsoft.Test.E2E.AspNet.OData.QueryComposition.SecurityTests.&lt;&gt;c__DisplayClass9_0.&lt;&lt;TestDosAttackWithMultipleThreads&gt;b__0&gt;d.MoveNext() in C:\BuildAgent\_work\1\s\test\E2ETest\Microsoft.Test.E2E.AspNet.OData\QueryComposition\SecurityTests.cs:line 95&#xD;
--- End of stack trace from previous location where exception was thrown ---&#xD;
   at System.Threading.Tasks.Task.&lt;&gt;c.&lt;ThrowAsync&gt;b__139_1(Object state)&#xD;
   at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()&#xD;
   at System.Threading.ThreadPoolWorkQueue.Dispatch()&#xD;
System.Threading.Tasks.TaskCanceledException: The operation was canceled.&#xD;
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)&#xD;
   at Microsoft.Test.E2E.AspNet.OData.QueryComposition.SecurityTests.TestDosAttack(String filter) in C:\BuildAgent\_work\1\s\test\E2ETest\Microsoft.Test.E2E.AspNet.OData\QueryComposition\SecurityTests.cs:line 85&#xD;
   at Microsoft.Test.E2E.AspNet.OData.QueryComposition.SecurityTests.&lt;&gt;c__DisplayClass9_0.&lt;&lt;TestDosAttackWithMultipleThreads&gt;b__0&gt;d.MoveNext() in C:\BuildAgent\_work\1\s\test\E2ETest\Microsoft.Test.E2E.AspNet.OData\QueryComposition\SecurityTests.cs:line 95&#xD;
--- End of stack trace from previous location where exception was thrown ---&#xD;
   at System.Threading.Tasks.Task.&lt;&gt;c.&lt;ThrowAsync&gt;b__139_1(Object state)&#xD;
   at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()&#xD;
   at System.Threading.ThreadPoolWorkQueue.Dispatch()&#xD;
</Text>
      </RunInfo>
    </RunInfos>

Environment

Please share additional details about the test environment.

only installed VS2019 Enterprise.

Additional Info

If i run the failure test case individually, it can pass:

as:

The text was updated successfully, but these errors were encountered:

singhsarab · 2019-12-23T16:15:35Z

@xuzhg I am not able to reproduce the issue. I followed the steps you have shared.

MarioAtTrimble · 2020-02-11T17:40:08Z

We are getting this issue from time to time on our build server, too. It's not consistent and sometimes it works just fine. We didn't see before when we were running core 2.2, but now see when we are running 3.1

nohwnd · 2020-02-13T13:04:51Z

We were seeing similar issues in VS, there was a problem with cancellation of run that produced a cancellation error and was not caught in the correct place. It resulted in the same error, but probably not the same stack. That error is kind of a catch all for all errors when the other process disconnects in the middle of an operation so it is hard to tell. I would suggest upgrading to 16.5.0 to see if that was fixes.

adaughterson · 2020-03-05T18:47:03Z

We are using vstest indirectly (via Microsoft.NET.Test.Sdk), and I have upgraded it to 16.5.0, and yet the error persists.

> A total of 1 test files matched the specified pattern.
> The active test run was aborted. Reason: Test host process crashed : Fatal error. Internal CLR error. (0x80131506)
>    at System.Reflection.RuntimeModule.GetTypes(System.Reflection.RuntimeModule)
>    at System.Reflection.RuntimeAssembly.get_DefinedTypes()
>    at Xunit.Runner.VisualStudio.VsTestRunner.GetAvailableRunnerReporters(System.Collections.Generic.IEnumerable`1<System.String>)
>    at Xunit.Runner.VisualStudio.VsTestRunner+<>c__DisplayClass23_0.<GetRunnerReporter>b__0()
>    at System.Lazy`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ViaFactory(System.Threading.LazyThreadSafetyMode)
>    at System.Lazy`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ExecutionAndPublication(System.LazyHelper, Boolean)
>    at System.Lazy`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].CreateValue()
>    at System.Lazy`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].get_Value()
>    at Xunit.Runner.VisualStudio.VsTestRunner.GetRunnerReporter(LoggerHelper, Xunit.Runner.VisualStudio.RunSettings, System.Collections.Generic.IEnumerable`1<System.String>)
>    at Xunit.Runner.VisualStudio.VsTestRunner.RunTests(Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle, LoggerHelper, Xunit.Runner.VisualStudio.TestPlatformContext, Xunit.Runner.VisualStudio.RunSettings, System.Func`1<System.Collections.Generic.List`1<Xunit.Runner.VisualStudio.AssemblyRunInfo>>)
>    at Xunit.Runner.VisualStudio.VsTestRunner.Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.ITestExecutor.RunTests(System.Collections.Generic.IEnumerable`1<System.String>, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle)
>    at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.Utilities.LazyExtension`2<Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.ITestExecutor,Microsoft.VisualStudio.TestPlatform.Common.Interfaces.ITestExecutorCapabilities>, System.Tuple`2<System.Uri,System.String>, Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Adapter.RunContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle)
>    at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(System.Collections.Generic.IEnumerable`1<System.Tuple`2<System.Uri,System.String>>, Int64)
>    at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestsInternal()
>    at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTests()
>    at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.ExecutionManager.StartTestRun(System.Collections.Generic.Dictionary`2<System.String,System.Collections.Generic.IEnumerable`1<System.String>>, System.String, System.String, Microsoft.VisualStudio.TestPlatform.ObjectModel.Engine.ClientProtocol.TestExecutionContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Engine.ITestCaseEventsHandler, Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.ITestRunEventsHandler)
>    at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TestRequestHandler+<>c__DisplayClass30_3.<OnMessageReceived>b__3()
>    at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TestRequestHandler+<>c.<.ctor>b__18_1(System.Action)
>    at Microsoft.VisualStudio.TestPlatform.Utilities.JobQueue`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SafeProcessJob(System.__Canon)
>    at Microsoft.VisualStudio.TestPlatform.Utilities.JobQueue`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].BackgroundJobProcessor()
>    at Microsoft.VisualStudio.TestPlatform.Utilities.JobQueue`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<.ctor>b__12_0()
>    at System.Threading.Tasks.Task.InnerInvoke()
>    at System.Threading.Tasks.Task+<>c.<.cctor>b__274_0(System.Object)
>    at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
>    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
>    at System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread)
>    at System.Threading.Tasks.ThreadPoolTaskScheduler+<>c.<.cctor>b__10_0(System.Object)
>    at System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
>    at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
>    at System.Threading.ThreadHelper.ThreadStart(System.Object)

Are there any other recommendations?

gregpakes · 2020-06-16T22:28:29Z

Our builds have started failing with this error. Is there a way to get some decent logs out? We are using azure hosted pipelines.

We have been getting this error for months but it's been very intermittent, so hasn't bothered us. But over the past 2 days it has spiked hugely.

amoerie · 2020-07-11T15:12:49Z

I'm also seeing persistent test host crashes in a project that I contribute to.

Links:

https://github.com/fo-dicom/fo-dicom/runs/861061964
https://github.com/fo-dicom/fo-dicom/runs/857136803
https://github.com/fo-dicom/fo-dicom/runs/857086214 (this was an internal CLR error)

ChaosEngine · 2020-07-29T19:58:27Z

In follow up to this I've created #2501

jesperoastrom · 2020-10-20T13:50:55Z

We had the exact same issue with our build. It turned out to be misconfigured settings for code coverage. In our CodeCoverage.runsettings-file we were not properly excluding test projects from coverage.

mcallaghan-bsm · 2020-10-27T20:27:48Z

We have (possibly) the same intermittent issue here.

rough important list of packages in our UT test project:

        <PackageReference Include="Microsoft.NETCore.Platforms" Version="3.1.0" />
        <PackageReference Include="coverlet.collector" Version="1.3.0">
        <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0" />
        <PackageReference Include="xunit" Version="2.4.1" />
        <PackageReference Include="xunit.runner.visualstudio" Version="2.4.1" />

Our default gitlab runner job is on a kubernetes runner, and defaults to docker image: "mcr.microsoft.com/dotnet/core/sdk:3.1-alpine"

Adding diag to a parallel job to see if we can repro with more diagnostics.

plachta11b · 2020-10-30T13:30:59Z

I have got here due to use of wrong function.
Assert.ThrowsException<MyException>(async () => await myFunction())
Fixed by replacing to async version:
await Assert.ThrowsExceptionAsync<MyException>(async () => await myFunction())

chaoticsoftware · 2020-11-09T21:55:11Z

Anyone has any update for this issue or any idea how to solve it?

ghost · 2021-01-14T14:33:04Z

I am seeing the same error message both with vstest.console.exe on Visual Studio 2019 16.8.4 and with dotnet test.
My test projects are using MSTest V2 v2.1.0. On the Windows 10 machine (OS build 17763.1697) I have installed .NET framework 4.7.2.

dotnet test MyMsTestProject.dll --filter OneSpecificTest -v diag
Microsoft (R) Test Execution Command Line Tool Version 16.8.3
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
The active test run was aborted. Reason: Test host process crashed : Process is terminated due to StackOverflowException.


Test Run Aborted.

With dotnet test I get this error in the Application event log:

Faulting application name: testhost.net472.x86.exe, version: 15.0.0.0, time stamp: 0xd89b8ef9
Faulting module name: KERNELBASE.dll, version: 10.0.17763.1697, time stamp: 0x672c12bb
Exception code: 0xc00000fd
Fault offset: 0x00108f0d
Faulting process id: 0x1b38
Faulting application start time: 0x01d6ea7f68419dc1
Faulting application path: C:\Program Files\dotnet\sdk\5.0.102\TestHost\testhost.net472.x86.exe
Faulting module path: C:\Windows\System32\KERNELBASE.dll
Report Id: 25d2099b-1d8a-4847-a90d-680e65d12f6e
Faulting package full name:
Faulting package-relative application ID:

With vstest.console.exe this error is logged:

Faulting application name: testhost.net472.exe, version: 15.0.0.0, time stamp: 0x89f6cab7
Faulting module name: clr.dll, version: 4.7.3740.0, time stamp: 0x5f7e5a1b
Exception code: 0xc00000fd
Fault offset: 0x000000000005c71b
Faulting process id: 0x1a70
Faulting application start time: 0x01d6ea818c7cd9ec
Faulting application path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\Extensions\TestPlatform\testhost.net472.exe
Faulting module path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Report Id: 62f466eb-cece-41ac-a934-cf21ea4080c8
Faulting package full name:
Faulting package-relative application ID:

A TestResults directory was created next to the test assembly, but it is completely empty.

Any idea how I can find the cause of the error?

nohwnd · 2021-01-14T14:53:02Z

@dbtfsbre you can install 5.0.101 dotnet SDK, and also install ProcDump so it is available in your PATH (if you open command line and type procdump, you should see output of the procdump tool). Then you do just:

dotnet test MyMsTestProject.dll --filter OneSpecificTest -v diag --blame-crash

And it should create a dump for you, open that in VS and you should see where that exception is coming from. Most likely it is your code calling some method recursively and forgetting to terminate.

ghost · 2021-01-14T16:56:57Z

@nohwnd Thanks. Something went wrong when copying files to the test machine for an investigation. The test is running fine in Visual Studio using Test Explorer and during our regular Release test runs. After compiling the test again directly on the test machine, I was able to run it with dotnet test as well as vstest.console.exe.

mcallaghan-geotab · 2021-03-06T01:09:35Z

FWIW: the solution we found was actually due to OOM kills from kubernetes gitlab-runner setup. The nodes were stretching their RAM too tightly, and eventually digging showed events where k8s was killing the dotnet process.

Looked like this:

Memory cgroup out of memory: Killed process 3010546 (dotnet) total-vm:5186748kB, anon-rss:378116kB, file-rss:67156kB, shmem-rss:0kB, UID:0 pgtables:1336kB oom_score_adj:936

This might not be the problem or solution for everyone, but wanted to share back as it is definitely something to consider!

felixntny · 2021-10-22T07:32:53Z

I'm having a similar problem. My actual test code, however, was a recursive function that would not exit the loop due to my test data. It also results in an infinite cycle. I adjusted my data to get out of the recursion function, and it worked.

rkieslinger · 2021-12-07T11:53:10Z

We had the exact same issue with our build. It turned out to be misconfigured settings for code coverage. In our CodeCoverage.runsettings-file we were not properly excluding test projects from coverage.

@jesperoastrom We have the same issue and it seems it is occurring since we re-activated code coverage. Can you please explain a little further, how you exactly solved your problem?

jesperoastrom · 2021-12-08T07:48:34Z

@rkieslinger If i remember correctly, we were running the test command and collecting coverage using the arguments: --collect "Code Coverage" --settings <SomePath>/CodeCoverage.runsettings. In the CodeCoverage.runsettings file we had misconfigured the exclusion of test modules/assemblies which caused this issue for us. We solved the problem by correctly exluding our test projects from code coverage using the CodeCoverage.runsettings file in ways described here: https://docs.microsoft.com/en-us/visualstudio/test/customizing-code-coverage-analysis#include-or-exclude-assemblies-and-members.

demonccc · 2021-12-17T00:05:52Z

We have (possibly) the same intermittent issue here.

rough important list of packages in our UT test project:
        <PackageReference Include="Microsoft.NETCore.Platforms" Version="3.1.0" />
        <PackageReference Include="coverlet.collector" Version="1.3.0">
        <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0" />
        <PackageReference Include="xunit" Version="2.4.1" />
        <PackageReference Include="xunit.runner.visualstudio" Version="2.4.1" />
Our default gitlab runner job is on a kubernetes runner, and defaults to docker image: "mcr.microsoft.com/dotnet/core/sdk:3.1-alpine"

Adding diag to a parallel job to see if we can repro with more diagnostics.

Hi, have you fixed this issue? We have a similar problem with dotnet and gitlab + k8s.

mcallaghan-geotab · 2021-12-17T20:39:50Z

@demonccc - see #2261 (comment)

demonccc · 2022-01-03T20:12:39Z

@demonccc - see #2261 (comment)

Thanks for the response. Could you tell me what configuration have you changed in order to avoid de OOM killer? The k8s executor of Gitlab doesn't have this option, and you can't modify the OOM adj of kubernetes.

Thanks in advance.

itaibh · 2022-01-05T11:58:13Z

It happens to me too when converting a project to use .net 5. We use NUnit and I made sure I use all the latest versions of the libraries. Out of more than 700 tests in one of the suites, 7 throw an exception such as this which crashes the test host process. Before changing to .net 5 (from .net framework 4.6.1, which means many changes were required but in those specific tests the flow seems untouched) everything worked.

I debugged the old version and the new version simultaneously and I see the same exception getting thrown, but in the .net framework it's getting handled correctly and the test passes while in .net 5 the exception isn't handled by the test host process and it crashes.

The test itself is quite short but hides complexities regarding threads, which might shed some light on where the problem is. Since the old code made use of Thread.Abort() which was deprecated in favor of Task.Cancel(), we had to replace Threads with Tasks, which might also narrow the search for what the root cause of the problem is.

itaibh · 2022-01-05T14:27:58Z

Okay, I now have a reproduction for the problem, and it's thread related.

This is the error I get:

========== Starting test run ==========
NUnit Adapter 4.2.0.0: Test execution started
Running selected tests in C:\Users\itaib\source\repos\TestHostCrashTest\TestHostCrashTest\bin\Debug\net5.0\TestHostCrashTest.dll
   NUnit3TestExecutor discovered 1 of 1 NUnit test cases using Current Discovery mode, Non-Explicit run
The active test run was aborted. Reason: Test host process crashed : Unhandled exception. System.Exception: some value
   at TestHostCrashTest.CrashTest.<>c.<CrashTestHost>b__0_0() in C:\Users\itaib\source\repos\TestHostCrashTest\TestHostCrashTest\CrashTest.cs:line 14
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

========== Test run aborted: 0 Tests (0 Passed, 0 Failed, 0 Skipped) run in < 1 ms ==========

and this is the code:

using NUnit.Framework;
using System.Threading;

namespace TestHostCrashTest
{
    public class CrashTest
    {
        [Test]
        public void CrashTestHost()
        {
            var t = new Thread(() =>
                throw new Exception("some value")
            );

            t.IsBackground = true;
            t.Start();
            t.Join();
        }
    }
}

juwens · 2022-01-17T16:18:02Z

It seems, that this is the default behavior of the .net runtime (except 1.0 and 1.1; see legacyUnhandledExceptionPolicy ). I didn't knew that either.
EVERY crashing Thread will terminate the application, even without a join.

There's not much vstest can do about it, i guess.

For very special scenarios you can use the mentioned flag in your app.config. This will prevent the app from terminating when an unhandled exception happens in a thread.

example app.config

<configuration>
   <runtime>
      <legacyUnhandledExceptionPolicy enabled="1"/>
   </runtime>
</configuration>

itaibh · 2022-01-17T19:23:31Z

@juwens Then how do you explain it works on .net 4.6? It's a different runtime... There's a specific problem introduced in .net core and above.

juwens · 2022-01-18T00:19:15Z

@juwens Then how do you explain it works on .net 4.6? It's a different runtime... There's a specific problem introduced in .net core and above.

just tried it with .net 4.5 target framework and it terminates the app too. Might be because clr tries to run code on the latest installed framework. which is 4.8 in my case.

my code:

using System;
using System.Threading;

namespace ConsoleAppNetFx
{
    internal class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("start netfx app");

            var t = new Thread(() =>
            {
                throw new Exception("exception in thread");
            });

            t.Start();
            
            while (true)
            {
                Console.WriteLine("sleep");
                Thread.Sleep(100);
            }

            Console.WriteLine("happy end");
        }
    }
}```

itaibh · 2022-01-18T08:15:53Z

@juwens Yes, of course it breaks a PROGRAM, but we're discussing a UNIT TEST

juwens · 2022-01-18T17:09:01Z

@juwens Yes, of course it breaks a PROGRAM, but we're discussing a UNIT TEST

are you aware, that a running test is just .net program (test host) which invokes your unit-test methods? 🤔
Of course it behaves the identical to a programm.
There is no magic in the test host.
Have you tried setting legacyUnhandledExceptionPolicy ?

itaibh · 2022-01-18T18:24:45Z

Yes, of course I'm aware of that. Are you aware that this test host is dynamically loading the tests from a DLL file? If it worked before then there's no reason it'll break now, which means this test host isn't doing it's job well. And I didn't try that since it's irrelevant for .net core and up, which doesn't even read app.config files anymore. Besides, setting the tests to run as STA isn't a good option since the code itself is multithreaded and we want to test in parallel as much as possible.

…

On Tue, Jan 18, 2022, 19:09 Jens ***@***.***> wrote: @juwens <https://github.com/juwens> Yes, of course it breaks a PROGRAM, but we're discussing a UNIT TEST are you aware, that a running test is just .net program (test host) which invokes your unit-test methods? 🤔 Of course it behaves the identical to a programm. There is no magic in the test host. Have you tried setting legacyUnhandledExceptionPolicy ? — Reply to this email directly, view it on GitHub <#2261 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABCNNYAWGXDILPNP5N6VHTUWWNDRANCNFSM4JU4X5FQ> . You are receiving this because you commented.Message ID: ***@***.***>

brah-mcdude · 2022-02-09T17:33:01Z

I had this issue. Here is how I resolved it:

I read the test logs (output | Tests tab).
I looked at the call-stack where the logs were saying that the process crashed.
It was an async void method. Obviously my code was causing the crash. Switched to async Task. Issue resolved.
But that was in my specific case. It should be something as simple as that in your specific case - in other words - your code is causing it.
If unable to resolve - post your Output | Tests somewhere and publish a link to it here.
We will help you figure it out. :)

brah-mcdude · 2022-02-09T17:35:34Z

I recommend closing this issue.

mcallaghan-geotab · 2022-02-09T18:13:41Z

@brah-mcdude - if the application/test code itself caused the crash, this should be much more obvious/different from the other scenarios presented "The active test run was aborted. Reason: Test host process crashed" <-- too generic.

brah-mcdude · 2022-02-09T19:05:01Z

@mcallaghan-geotab - Here is the message I was getting: "The active test run was aborted. Reason: Test host process crashed". My code caused it. I traced the call stack. Show me your code. I will show you the cause of this issue.

cn-ml · 2022-02-10T17:09:52Z

I just got a similar problem with my application in .NET 6

The active test run was aborted. Reason: Test host process crashed : Stack overflow.
Repeat 19163 times:
--------------------------------
   at Studip.NET.EntityHandlerBase`1[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].get_BaseUrl()
--------------------------------

Edit:
I just removed some lines of code including attributes/reflection and now the test runs perfectly. I don't know why that caused it and how to fix it.
This is the line of code I removed:

typeof(TEntity)
        .GetCustomAttributes(typeof(SchemaNameAttribute), false)
        .OfType<SchemaNameAttribute>()
        .FirstOrDefault()?.Type
        ?? throw new InvalidProgramException($"Type '{typeof(TEntity).FullName}' has no '{nameof(SchemaNameAttribute)}' to be used in type handler base!");

brah-mcdude · 2022-02-10T18:19:08Z

@cn-ml - you have a "Stack overflow". Show us your call-stack.

mamift · 2022-07-12T04:01:03Z

I've been getting this same issue, but it seemingly occurs only when running a whole test project at once; if I try to test each individual test methods in their own session/run then I never encounter the problem.

I also never get a stack trace or any meaningful debug output just before the Stack overflow occurs.

This is the most I've been able to extract out of the test/debug output:

========== Starting test run ==========
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.3+1b45f5407b (64-bit .NET Core 3.1.26)
[xUnit.net 00:00:02.50]   Starting:    McrApi.Tests.Integration
The active test run was aborted. Reason: Test host process crashed : Stack overflow.

Running/debugging the test project seems to make no difference. I've never been able to get VS to even break on the Stack overflow; this is just bizarre.

Using Xunit/.NET Core 3.1.

brah-mcdude · 2022-07-13T04:58:32Z

"I've been getting this same issue, but it seemingly occurs only when running a whole test project at once; if I try to test each individual test methods in their own session/run then I never encounter the problem."

multiple tests are stress loading each other. try running each test with a cpu-stress app. https://learn.microsoft.com/en-us/sysinternals/downloads/cpustres.

or, maybe your tests are sharing data. maybe you have static (global) variables.
scenario: test1 writes to global static variable1, test2 overwrites variable1. test1 reads a corrupted value from variable1.

mamift · 2022-07-13T05:29:17Z

maybe your tests are sharing data. maybe you have static (global) variables. scenario: test1 writes to global static variable1, test2 overwrites variable1. test1 reads a corrupted value from variable1.

Yeah I do have shared static variables, but their values are set in a static constructor and are read only.

What's bizarre here is the stack overflow error only occurs when I run the tests in VS or using the dotnet command. If I debug the tests in VS then it NEVER occurs. I can break in VS on other exceptions like ArgumentNullException or NullRef exceptions, but never stack overflow.

brah-mcdude · 2022-07-14T02:40:00Z

@mamift - having inconsistent test results is an indication of data corruption. is your static ctor invoked only once for ALL the tests? i would get rid of any static variables. each test should live in its own bubble universe. you are violating this sacred principle.

brah-mcdude · 2022-07-16T16:52:15Z

@mamift - if you have a multi-threaded code under test - that could also lead to inconsistent test results. but you would be able to reproduce the inconsistency in the test results even when you run a single test. so that is not a prime suspect.

[edit] use a cpu-stress-app https://learn.microsoft.com/en-us/sysinternals/downloads/cpustres. load up all cpu's to 100%. then run each test separately.

srothkin · 2022-09-30T20:53:45Z

After the most recent Visual Studio 2022 update I've started seeing this nearly constantly in one of our solutions (with 421 tests using xUnit). Using .NET 6. I've gone over this entire thread as well as some other discussions I've found. None lead to fixes. When I do get it to run without this error, there's only 1 failure (and that's a not implemented exception for a test I've yet to write).

juwens · 2022-09-30T21:07:54Z

After the most recent Visual Studio 2022 update I've started seeing this nearly constantly in one of our solutions (with 421 tests using xUnit). Using .NET 6. I've gone over this entire thread as well as some other discussions I've found. None lead to fixes. When I do get it to run without this error, there's only 1 failure (and that's a not implemented exception for a test I've yet to write).

try dotnet test —-blame it will reveal the source of the issue. In our case it were unhandled exceptions in async void methods and more tricky: async void lambdas.

srothkin · 2022-09-30T21:29:48Z

After the most recent Visual Studio 2022 update I've started seeing this nearly constantly in one of our solutions (with 421 tests using xUnit). Using .NET 6. I've gone over this entire thread as well as some other discussions I've found. None lead to fixes. When I do get it to run without this error, there's only 1 failure (and that's a not implemented exception for a test I've yet to write).

try dotnet test —-blame it will reveal the source of the issue. In our case it were unhandled exceptions in async void methods and more tricky async void lambdas.

How do I get Visual Studio to do that with the test explorer runner?

JBoothUA · 2022-10-14T14:23:15Z

just in case this helps someone, at a random time this line started randomly hanging our tests

_task = Task.Factory.StartNew(async () =>
{
await Task.Delay((int)cleanInterval.TotalMilliseconds, _cancellationTokenSource.Token);
--> while (!_cancellationTokenSource.Token.IsCancellationRequested)

brah-mcdude · 2022-10-15T14:44:44Z

I just took a look at the original issue's description. #2261 (comment)
I looked at the crash call stack. And I noticed: TestDosAttackWithMultipleThreads.
I would like to take a look at the code. Threads are very difficult to get right.

brah-mcdude · 2022-10-15T15:25:15Z

After the most recent Visual Studio 2022 update I've started seeing this nearly constantly in one of our solutions (with 421 tests using xUnit). Using .NET 6. I've gone over this entire thread as well as some other discussions I've found. None lead to fixes. When I do get it to run without this error, there's only 1 failure (and that's a not implemented exception for a test I've yet to write).

try dotnet test —-blame it will reveal the source of the issue. In our case it were unhandled exceptions in async void methods and more tricky async void lambdas.

That is literally what I wrote above as the cause of crash in my case (#2261 (comment)).
@xuzhg - please search for "async void" in your code.

brah-mcdude · 2022-10-25T03:08:38Z

just in case this helps someone, at a random time this line started randomly hanging our tests

_task = Task.Factory.StartNew(async () => { await Task.Delay((int)cleanInterval.TotalMilliseconds, _cancellationTokenSource.Token); --> while (!_cancellationTokenSource.Token.IsCancellationRequested)

@JBoothUA - With all due respect. your hanging line seems irrelevant to this issue. Not that this issue is actually an issue (I keep saying that there is no issue and we should close it). But your issue is even simpler than this issue. Both your issue and this issue seem to emanate from your (and @xuzhg's) bad threading code.

jakubkrzysztofsikora · 2023-01-19T14:39:31Z

After the most recent Visual Studio 2022 update I've started seeing this nearly constantly in one of our solutions (with 421 tests using xUnit). Using .NET 6. I've gone over this entire thread as well as some other discussions I've found. None lead to fixes. When I do get it to run without this error, there's only 1 failure (and that's a not implemented exception for a test I've yet to write).

try dotnet test —-blame it will reveal the source of the issue. In our case it were unhandled exceptions in async void methods and more tricky async void lambdas.

That is literally what I wrote above as the cause of crash in my case (#2261 (comment)). @xuzhg - please search for "async void" in your code.

Holy moly, that was it. Thanks dude.

brah-mcdude · 2023-01-26T20:37:39Z

@xuzhg can you please close this issue? Reason: CANNOT REPRODUCE.

gregpakes mentioned this issue Jun 17, 2020

VSTest repeatedly fails #2464

Closed

linkdotnet mentioned this issue Apr 28, 2022

Blame when running tests might hide bugs? bUnit-dev/bUnit#694

Closed

nohwnd assigned nohwnd and unassigned nohwnd Jul 13, 2022

joshmar mentioned this issue Nov 16, 2022

[Question] test run aborted. microsoft/playwright-dotnet#2386

Closed

Evangelink added bug needs-triage This item should be discussed in the next triage meeting. labels Nov 29, 2022

Evangelink removed bug needs-triage This item should be discussed in the next triage meeting. labels Jan 27, 2023

Evangelink closed this as completed Jan 27, 2023

seantleonard mentioned this issue Mar 5, 2024

[Bug]: MSSQL (primarily Linux) pipeline flakey Azure/data-api-builder#2010

Open

1 task

The active test run was aborted. Reason: Test host process crashed #2261

The active test run was aborted. Reason: Test host process crashed #2261

Comments

xuzhg commented Dec 3, 2019

Description

Steps to reproduce

Expected behavior

Actual behavior

Environment

Additional Info

singhsarab commented Dec 23, 2019

MarioAtTrimble commented Feb 11, 2020

nohwnd commented Feb 13, 2020

adaughterson commented Mar 5, 2020

gregpakes commented Jun 16, 2020

amoerie commented Jul 11, 2020

ChaosEngine commented Jul 29, 2020 • edited Loading

jesperoastrom commented Oct 20, 2020

mcallaghan-bsm commented Oct 27, 2020 • edited Loading

plachta11b commented Oct 30, 2020

chaoticsoftware commented Nov 9, 2020

ghost commented Jan 14, 2021 • edited by ghost Loading

nohwnd commented Jan 14, 2021

ghost commented Jan 14, 2021

mcallaghan-geotab commented Mar 6, 2021

felixntny commented Oct 22, 2021

rkieslinger commented Dec 7, 2021 • edited Loading

jesperoastrom commented Dec 8, 2021 • edited Loading

demonccc commented Dec 17, 2021

mcallaghan-geotab commented Dec 17, 2021

demonccc commented Jan 3, 2022

itaibh commented Jan 5, 2022

itaibh commented Jan 5, 2022 • edited Loading

juwens commented Jan 17, 2022 • edited Loading

itaibh commented Jan 17, 2022

juwens commented Jan 18, 2022 • edited Loading

itaibh commented Jan 18, 2022

juwens commented Jan 18, 2022

itaibh commented Jan 18, 2022 via email

brah-mcdude commented Feb 9, 2022

brah-mcdude commented Feb 9, 2022

mcallaghan-geotab commented Feb 9, 2022

brah-mcdude commented Feb 9, 2022

cn-ml commented Feb 10, 2022 • edited Loading

brah-mcdude commented Feb 10, 2022 • edited Loading

mamift commented Jul 12, 2022

brah-mcdude commented Jul 13, 2022 • edited Loading

mamift commented Jul 13, 2022

brah-mcdude commented Jul 14, 2022 • edited Loading

brah-mcdude commented Jul 16, 2022 • edited Loading

srothkin commented Sep 30, 2022

juwens commented Sep 30, 2022 • edited Loading

srothkin commented Sep 30, 2022

JBoothUA commented Oct 14, 2022

brah-mcdude commented Oct 15, 2022

brah-mcdude commented Oct 15, 2022 • edited Loading

brah-mcdude commented Oct 25, 2022

jakubkrzysztofsikora commented Jan 19, 2023

brah-mcdude commented Jan 26, 2023

ChaosEngine commented Jul 29, 2020 •

edited

Loading

mcallaghan-bsm commented Oct 27, 2020 •

edited

Loading

ghost commented Jan 14, 2021 •

edited by ghost

Loading

rkieslinger commented Dec 7, 2021 •

edited

Loading

jesperoastrom commented Dec 8, 2021 •

edited

Loading

itaibh commented Jan 5, 2022 •

edited

Loading

juwens commented Jan 17, 2022 •

edited

Loading

juwens commented Jan 18, 2022 •

edited

Loading

cn-ml commented Feb 10, 2022 •

edited

Loading

brah-mcdude commented Feb 10, 2022 •

edited

Loading

brah-mcdude commented Jul 13, 2022 •

edited

Loading

brah-mcdude commented Jul 14, 2022 •

edited

Loading

brah-mcdude commented Jul 16, 2022 •

edited

Loading

juwens commented Sep 30, 2022 •

edited

Loading

brah-mcdude commented Oct 15, 2022 •

edited

Loading