mstest 3.4.0 cannot work with .net 472 when running from Visual Studio 17.10.0 ?

[fforjan]

Describe the bug

Trying to run a hello world unit tests with .net 472 results in :

[5/23/2024 7:11:13.805 AM] [Error] An exception occurred while invoking executor 'executor://mstestadapter/v2': Could not load file or assembly 'Microsoft.VisualStudio.TestPlatform.TestFramework.Extensions, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
Stack trace:
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestDeployment.GetDeploymentInformation(String source)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.ExecuteTestsInSource(IEnumerable`1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle, String source, Boolean isDeploymentDone) in /_/src/Adapter/MSTest.TestAdapter/Execution/TestExecutionManager.cs:line 238
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.ExecuteTests(IEnumerable`1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle, Boolean isDeploymentDone) in /_/src/Adapter/MSTest.TestAdapter/Execution/TestExecutionManager.cs:line 146
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.RunTests(IEnumerable`1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle, TestRunCancellationToken runCancellationToken) in /_/src/Adapter/MSTest.TestAdapter/Execution/TestExecutionManager.cs:line 85
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestExecutor.<>c__DisplayClass8_0.<RunTests>b__1(TestRunCancellationToken testRunToken) in /_/src/Adapter/MSTest.TestAdapter/VSTestAdapter/MSTestExecutor.cs:line 57
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestExecutor.<>c__DisplayClass11_0.<RunTestsFromRightContext>g__DoRunTests|0() in /_/src/Adapter/MSTest.TestAdapter/VSTestAdapter/MSTestExecutor.cs:line 126
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestExecutor.RunTestsFromRightContext(IFrameworkHandle frameworkHandle, Action`1 runTestsAction) in /_/src/Adapter/MSTest.TestAdapter/VSTestAdapter/MSTestExecutor.cs:line 134
   at Microsoft.VisualStudio.TestPlatform.Common.ExtensionDecorators.SerialTestRunDecorator.RunTests(IEnumerable`1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle) in /_/src/Microsoft.TestPlatform.Common/ExtensionDecorators/SerialTestRunDecorator.cs:line 46
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithTests.InvokeExecutor(LazyExtension`2 executor, Tuple`2 executorUri, RunContext runContext, IFrameworkHandle frameworkHandle) in /_/src/Microsoft.TestPlatform.CrossPlatEngine/Execution/RunTestsWithTests.cs:line 73
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.<>c__DisplayClass46_0.<RunTestInternalWithExecutors>b__0() in /_/src/Microsoft.TestPlatform.CrossPlatEngine/Execution/BaseRunTests.cs:line 464
   at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformThread.<>c__DisplayClass0_0.<Run>b__0() in /_/src/Microsoft.TestPlatform.PlatformAbstractions/net462/System/PlatformThread.cs:line 29
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformThread.Run(Action action, PlatformApartmentState apartmentState, Boolean waitForCompletion) in /_/src/Microsoft.TestPlatform.PlatformAbstractions/net462/System/PlatformThread.cs:line 47
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.TryToRunInStaThread(Action action, Boolean waitForCompletion) in /_/src/Microsoft.TestPlatform.CrossPlatEngine/Execution/BaseRunTests.cs:line 680
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable`1 executorUriExtensionMap, Int64 totalTests) in /_/src/Microsoft.TestPlatform.CrossPlatEngine/Execution/BaseRunTests.cs:line 464

Steps To Reproduce

Create a new unit test projects based on the sdk format.
Update the property into the project to :

 <PropertyGroup>
   <TargetFrameworks>net472</TargetFrameworks>
   <ImplicitUsings>enable</ImplicitUsings>
   <Nullable>enable</Nullable>
  <LangVersion>12</LangVersion>
   <IsPackable>false</IsPackable>
   <IsTestProject>true</IsTestProject>
 </PropertyGroup>

 <ItemGroup>
  <PackageReference Include="coverlet.collector" Version="6.0.0" />
  <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.10.0" />
  <PackageReference Include="MSTest.TestAdapter" Version="3.4.0" />
  <PackageReference Include="MSTest.TestFramework" Version="3.4.0" />
</ItemGroup>

(notice the target framework as net472)

Expected behavior

Tests should run when executed from visual studio

Actual behavior

Tests are not run

Additional context

Executed within
Microsoft Visual Studio Enterprise 2022 (64-bit) - Current
Version 17.10.0

if run in command line:

TestProject1\bin\Debug\net472>vstest.console TestProject1.dll
Microsoft (R) Test Execution Command Line Tool Version 17.10.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
  Passed TestMethod1 [4 ms]

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 0.3981 Seconds

AB#2065771

[Evangelink]

Do you have any other project within the solution? It looks exactly like the issue that happens when running tests with VSTest and having a mix of MSTest v2 and MSTest v3 projects (see microsoft/vstest#3475).

[fforjan]

No brand new solution/unit test from scratch from VS2022

[Evangelink]

I confirm that I can reproduce, I will investigate and work on a fix.

[testplatform-bot]

❌ There was a problem linking to Azure Boards work item(s):

• 2065771

Please check the IDs and try again using the AB# syntax. Learn more

[flensrocker]

Same error here with net48.

[Evangelink]

Update: all .NET fx versions are broken, using .NET core or using MSTest runner there is no issue. So something is conflicting/clashing between MSTest and VSTest.

[flensrocker]

0 days since binding redirects? 🍻

Staying on 3.3.1 is not a problem for now.

Thank you!

[Evangelink]

0 days since binding redirects? 🍻

Not a binding redirect issue, it's a custom assembly resolver issue. Don't ask me why .NET FX was designed with so much complexity.

[Evangelink]

So far no good news. We have confirmed the issue is not coming from VSTest alone nor MSTest alone but a combination of both. The difference of running v3.3.1 vs v3.4.0 is that in v3.3.1 MSTest custom assembly resolver kicks in and resolves the assembly which isn't the case with v3.4.0. Sadly, I don't find any change that would be obvious for causing this issue...

I am trying to find the commit that started to introduce the failure.

I will be posting updates when possible.

[tscilipoti]

Same issue running VS 2022 targeting .NET Framework 4.7.2. C#. Reverting to 3.3.1 resolves.
(Microsoft Visual Studio Enterprise 2022 (64-bit) - Current Version 17.9.6)

[flensrocker]

#2948
For real backward compatibility I do #if more than I want. 🤷‍♂️

Thank you for finding the bug!

[Rans4ckeR]

3.4.0 and 3.4.1 are both not working while 3.3.1 was fine for this scenario:

• .NET8 ProjectX
• .NET8 test ProjectY which uses Microsoft Fakes for project X
• Build on Azure DevOps using VSBuild@1.
• Run ProjectY

The Fakes assemblies cannot be resolved, each test that uses them outputs:

System.IO.FileNotFoundException: Could not load file or assembly 'ProjectX.Fakes, Version=x.x.x.x, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified..

Where ProjectX.Fakes.dll is present in the output folder next to ProjectY.exe.

[Evangelink]

@Rans4ckeR I have moved your request as a separate issue as it's not related to original issue on this thread.

[flensrocker]

My tests are all green again after upgrading to 3.4.1 - thank you!

[Evangelink]

Awesome! Thanks for the confirmation @flensrocker.

@fforjan I hope it's good for you too?

[fforjan]

Yesterday was public holiday in US, will confirm later :)

[fforjan]

@Evangelink all good on our side !

[fforjan]

@Evangelink I talked to fast, i've successfully run a sanity check, but it seems one of our projects is not executed, but due to

[5/28/2024 11:35:20.679 AM] [Error] The active test run was aborted. Reason: Test host process crashed : Process is terminated due to StackOverflowException.

Which I think is potentially related to #2692 as @Evangelink you've commented

FYI @nohwnd

[fforjan]

Do you want a new issue ?

[Evangelink]

Yes please and if you could share diagnostic logs (see guide https://github.com/microsoft/vstest/blob/main/docs/diagnose.md) either on the issue or privately (my email or through VS feedback, see https://learn.microsoft.com/cpp/overview/how-to-report-a-problem-with-the-visual-cpp-toolset?view=msvc-170#to-create-a-problem-report-for-private-information)

[Evangelink]

Which I think is potentially related to #2692 as @Evangelink you've commented

This is fixed and only happening when resolving .resources.

[fforjan]

New workitem: #3019