Run dotnet nuget tests on multiple TFMs

[zivkan]

Bug

Fixes:NuGet/Home#9252
Regression: No

• Last working version:
• How are we preventing it in future:

Fix

Details:

• Add new TFM property for testing, and make Dotnet.Integration.Tests use it.
• Fix up test fixture to copy only one of the SDK folders for testing, the "nearest match" based on the runtime TFM and that SDK TFM.
• Create helper methods to write global.json in test folders, and change all tests in Dotnet.Integration.Tests to use it.

Testing/Validation

Tests Added: No
Reason for not adding tests: no product code changes, just modifying the test projects.
Validation:

[heng-liu]

There is one more test PackCommand_DoesNotGenerateOwnersElement at the end of PackCommandTests you need to change ;)
Not sure if it's the best practice to have all the tests changed to add the global.json file.
As it's easy to forget.
If in future someone want to add more tests into Dotnet.Integration.Tests, how do they know which method is the right method to use?(testDirectory = msbuildFixture.CreateTestDirectory() or testDirectory = TestDirectory.Create())

[zivkan]

Not sure if it's the best practice to have all the tests changed to add the global.json file.
As it's easy to forget.
If in future someone want to add more tests into Dotnet.Integration.Tests, how do they know which method is the right method to use?(testDirectory = msbuildFixture.CreateTestDirectory() or testDirectory = TestDirectory.Create())

I agree, from an API point of view it's really not good to have similar APIs.

However, TestDirectory.Create() is used by a whole lot of other projects, not just DotNet.Integration.Tests, and it doesn't make sense for any of them to write a global.json file, particularly as getting it wrong will cause tests to fail. These are the projects which I could quickly find usages: NuGet.CommandLine.FuncTest, NuGet.MSSigning.Extensions.FuncTest, NuGet.CommandLine.Test, NuGet.PackageManagement.UI.Test, NuGet.PackageManagement.VisualStudio.Test, NuGet.VisualStudio.Implementation.Test, NuGetConsole.Host.PowerShell.Test, NuGet.Commands.FuncTest, NuGet.Packaging.FuncTest, NuGet.Protocol.FuncTest, Microsoft.Build.NuGetSdkResolver.Test, NuGet.Build.Tasks.Console.Test, NuGet.Build.Tasks.Pack.Test, NuGet.Build.Tasks.Test, NuGet.Commands.Test, NuGet.Common.Test, NuGet.Configuration.Test, NuGet.PackageManagement.Test, NuGet.Packaging.Test, NuGet.ProjectModel.Test, and NuGet.Protocol.Tests.

The requirement to use global.json to lock a test to a specific SDK version is very specific to DotNet.Integration.Tests due to the fact that this is the only test project where we patch a custom .NET SDK folder and want to make sure we run only that SDK.

If you think it's better, I could create an overload, but developers would still need to know whether to call TestDirectory.Create() (everywhere except DotNet.Integration.Tests) or TestDirectory.Create(string) (only in DotNet.Integration.Tests).

Alternatively, we could investigate writing an analyzer that will error when TestDirectory.Create() is used within DotNet.Integration.Tests, and maybe even offer a codefix if it can find a MsbuildIntegrationTestFixture. But this would be a new issue, as I'd need to learn a lot, and therefore need a lot more time, to do it in this PR.

[nkolev92]

fwiw, I'm with @zivkan on the TestDirectory.Create stuff.

A question or two in the comments.

[heng-liu]

I agree that it's hard to find a "good place" ,as we use various APIs in dotnet integration tests.
I'm not sure if adding it in all the dotnet command methods in MsbuildIntegrationTestFixture will be better than this. (CreateDotnetNewProject, CreateDotnetToolProject,RestoreToolProject,RestoreProjectOrSolution,RunDotnet,PackProjectOrSolution,BuildProject and so on) I suppose all the dotnet integration test will call at least one dotnet command in MsbuildIntegrationTestFixture, but I'm not sure.
The pros is when adding new tests, we won't forget to do something.
The cons is it's not intuitive.

If the above is not as good as the original one, can you pls consider adding some comments on how to differentiate those method? Thanks.

[heng-liu]

The pros is when adding new tests, we won't forget to do something.

-----Just realized that, it's not true. If we need to add new dotnet command, we still need to remember adding global.json

[heng-liu]

Nice change! So no more adapting for future TFM change :)

[zivkan]

Unfortunately the template doesn't allow arbitrary TFMs. On my machine it only accepts net5.0, netstandard2.1 and netstandard2.0. So if a future SDK removes this TFM from the template, our tests will break.

[heng-liu]

I see..Thanks for clarification.

[heng-liu]

Good to know that we can use GetNearest to find the right bin/Debug/TFM folder, to patch the dotnet. So we can get rid of hardcoded TFM and the if statement in selecting TFM folder.

[zivkan]

We have multiple methods called GetNearest in several types, and I have no idea what the difference is between them, or if I'm using the "right" one, but it seems to work.

[kartheekp-ms]

I guess my PR to add couple of new dotnet tests is depending on this PR because one new test in my PR fails as NetStandard packages are not available in the local source created by SimpleTestPathContext.