Enable the fundamentals behind libraries tests compiled via crossgen2 #75230

davidwrighton · 2022-09-08T01:43:43Z

Document how to use the special modes of libraries compilation
Add the TestCrossgen2 flag to libraries testing
- Unlike TestNativeAot and TestSingleFile, use a self-contained strategy instead of actual single-file to improve debuggability of the produced output
Tweak the SingleFileTestRunner.cs to avoid the fork bomb problem introduced by RemoteTestExecutor in a self-contained, but not singlefile scenario
Add support for a set of flags to SingleFileTestRunner to respect the command line options I needed when debugging why the tests were not working.

- Document how to use the special modes of libraries compilation - Add the TestCrossgen2 flag to libraries testing - Unlike TestNativeAot and TestSingleFile, use a self-contained strategy instead of actual single-file to improve debuggability of the produced output - Tweak the SingleFileTestRunner.cs to avoid the fork bomb problem introduced by RemoteTestExecutor in a self-contained, but not singlefile scenario - Add support for a set of flags to SingleFileTestRunner to respect the command line options I needed when debugging why the tests were not working.

dotnet-issue-labeler · 2022-09-08T01:43:49Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

ghost · 2022-09-08T01:44:42Z

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

Document how to use the special modes of libraries compilation
Add the TestCrossgen2 flag to libraries testing
- Unlike TestNativeAot and TestSingleFile, use a self-contained strategy instead of actual single-file to improve debuggability of the produced output
Tweak the SingleFileTestRunner.cs to avoid the fork bomb problem introduced by RemoteTestExecutor in a self-contained, but not singlefile scenario
Add support for a set of flags to SingleFileTestRunner to respect the command line options I needed when debugging why the tests were not working.

Author:	davidwrighton
Assignees:	davidwrighton
Labels:	`area-Infrastructure-libraries`
Milestone:	-

davidwrighton · 2022-09-08T01:47:39Z

docs/workflow/testing/libraries/testing.md

+Each test project can potentially have multiple target frameworks. There are some tests that might be OS-specific, or might be testing an API that is available only on some target frameworks, so the `TargetFrameworks` property specifies the valid target frameworks.
+
+### Running tests in custom compilation modes


@ViktorHofer I'm interested in any particular details I missed here. We really need some documentation on this sort of thing, as the behaviors of our tests are not particularly clear to folks who don't look into these systems often.

MichalStrehovsky · 2022-09-08T05:20:12Z

src/libraries/Common/tests/SingleFileTestRunner/SingleFileTestRunner.cs

+                filters.ExcludedNamespaces.Add(args[i + 1].Trim());
+                i++;
+            }
+            if (args[i].Equals("-parallel", StringComparison.OrdinalIgnoreCase))


Thank you for extending this! Especially the -parallel option is something that I've been doing locally pretty much forever.

I would like us to get rid of this runner eventually (#70432). The xunit runner from Arcade used to support single file testing (including remote executor) but all of that was deleted when UWP support was deleted from Arcade. I'd like to restore it at some point.

I wonder though whether crossgen2 testing needs the single file runner - could it get by with the real runner?

I suspect it could use the normal file runner, but this looks like it will work, and I don't understand the libraries testing enough to understand how exactly to make this work with the normal runner without doing more research. With the additional arguments, I don't see a huge benefit to making this different from SingleFile or NativeAOT testing. As a side benefit, enabling R2R singlefile testing here becomes extremely easy should we want to do so.

MichalStrehovsky · 2022-09-08T05:22:21Z

eng/testing/tests.props

@@ -8,6 +8,7 @@
    <VSTestNoLogo>true</VSTestNoLogo>
    <ILLinkDescriptorsPath>$(MSBuildThisFileDirectory)ILLinkDescriptors\</ILLinkDescriptorsPath>
    <TestSingleFile Condition="'$(TestNativeAot)' == 'true'">true</TestSingleFile>
+    <TestSingleFile Condition="'$(TestCrossgen2)' == 'true'">true</TestSingleFile>


Would it make sense to just name this TestReadyToRun?

Yes. That's a better name

ivdiazsa · 2022-09-08T16:47:06Z

docs/workflow/testing/libraries/testing.md

+
+To run a test in specific mode, simply build the tests after building the prerequisite subsets, and specify the test mode on a command line such as:
+```cmd


NIT: Try to keep lists and code examples surrounded by blank lines, as per the recommended Markdown guidelines, so that we keep our md files as clear of warnings as we can.

ivdiazsa

Just the NIT I pointed out in the Markdown doc file, but otherwise looks good to me! Thanks for enabling this!

…ct references as content

trylek

LGTM, thank you!

MichalStrehovsky · 2022-09-09T02:19:39Z

eng/testing/tests.props

@@ -8,6 +8,7 @@
    <VSTestNoLogo>true</VSTestNoLogo>
    <ILLinkDescriptorsPath>$(MSBuildThisFileDirectory)ILLinkDescriptors\</ILLinkDescriptorsPath>
    <TestSingleFile Condition="'$(TestNativeAot)' == 'true'">true</TestSingleFile>
+    <TestSingleFile Condition="'$(TestReadyToRun)' == 'true'">true</TestSingleFile>


Not blocking because I guess it works, but ReadyToRun is not single file even if you squint really hard and having to opt out of a bunch of things in tests.singlefile.targets is now making things quite confusing. We'll likely have to add more hacks around it in the coming months/years. It's not great that we have to set it.

ViktorHofer · 2022-09-12T09:55:23Z

docs/workflow/testing/libraries/testing.md

+To run a test in specific mode, simply build the tests after building the prerequisite subsets, and specify the test mode on a command line such as:
+```cmd
+dotnet build /t:Test /p:Configuration=Release /p:TestReadyToRun=true


nit:

Suggested change

dotnet build /t:Test /p:Configuration=Release /p:TestReadyToRun=true

dotnet build /t:Test -c Release /p:TestReadyToRun=true

ViktorHofer · 2022-09-12T09:56:41Z

docs/workflow/testing/libraries/testing.md

+### Running tests in custom compilation modes
+
+There are several custom compilation modes for tests. These are enabled by setting a switch during the config.


What you mean by "during the config"? I assume you are talking about the test invocation? Might make sense to clarify that part.

ViktorHofer · 2022-09-12T09:57:38Z

src/libraries/Common/tests/SingleFileTestRunner/SingleFileTestRunner.cs

@@ -27,6 +27,9 @@ public static int Main(string[] args)
        var asm = typeof(SingleFileTestRunner).Assembly;
        Console.WriteLine("Running assembly:" + asm.FullName);

+        // The current RemoteExecutor implementation is not compatible with the SingleFileTestRunner
+        Environment.SetEnvironmentVariable("DOTNET_REMOTEEXECUTOR_SUPPORTED", "0");


Do we have a tracking issue for that, to link to?

ViktorHofer · 2022-09-12T09:59:26Z

src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj

+    <!-- R2R testing does not tolerate the combination of a regular project reference and a content reference. -->
+    <PublishReadyToRunExclude Include="LongPath.dll" />


Curious, why does R2R testing not allow a ProjectReference that contributes to the Content item? Is there a tracking issue for that?

ivdiazsa · 2022-10-13T20:38:02Z

Closing and reopening to get CI to run again and check the failures. It had been some time and they had been deleted.

davidwrighton · 2022-12-07T23:37:37Z

...sts/System.Diagnostics.FileVersionInfo.Tests/System.Diagnostics.FileVersionInfo.Tests.csproj

@@ -50,5 +50,7 @@
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </ProjectReference>
    <ProjectReference Include="..\System.Diagnostics.FileVersionInfo.TestAssembly\System.Diagnostics.FileVersionInfo.TestAssembly.csproj" Condition="'$(TargetOS)' == 'Browser'" />
+    <!-- R2R testing does not tolerate the combination of a regular project reference and a content reference. -->
+    <PublishReadyToRunExclude Include="System.Diagnostics.FileVersionInfo.TestAssembly.dll" />


This is actually a bug in the SDK which should be fixed there.

ivdiazsa · 2023-01-20T19:42:17Z

This small project is now taken over at PR #80946.

davidwrighton requested review from ViktorHofer, trylek and ivdiazsa September 8, 2022 01:43

ghost assigned davidwrighton Sep 8, 2022

davidwrighton added the area-Infrastructure-libraries label Sep 8, 2022

davidwrighton commented Sep 8, 2022

View reviewed changes

davidwrighton mentioned this pull request Sep 8, 2022

[Quality Week] Improve quality of Crossgen2 testing, most notably in context of library tests #74681

Closed

MichalStrehovsky reviewed Sep 8, 2022

View reviewed changes

Fix issues noted by code reviewers, and CI

6d7a7f3

ivdiazsa reviewed Sep 8, 2022

View reviewed changes

ivdiazsa approved these changes Sep 8, 2022

View reviewed changes

Actually use live built crossgen2, and workaround issues around proje…

4d9c0de

…ct references as content

trylek approved these changes Sep 8, 2022

View reviewed changes

MichalStrehovsky reviewed Sep 9, 2022

View reviewed changes

ViktorHofer reviewed Sep 12, 2022

View reviewed changes

ivdiazsa closed this Oct 13, 2022

ivdiazsa reopened this Oct 13, 2022

jakobbotsch mentioned this pull request Nov 7, 2022

Add ability for crossgen2 to synthesize PGO histograms #77683

Merged

davidwrighton commented Dec 7, 2022

View reviewed changes

ivdiazsa mentioned this pull request Jan 20, 2023

[Continuation] - Enable the fundamentals behind libraries tests compiled via Crossgen2 #80946

Merged

ivdiazsa closed this Jan 20, 2023

ghost locked as resolved and limited conversation to collaborators Feb 20, 2023

davidwrighton deleted the enable_r2r_libraries_tests branch April 13, 2023 18:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable the fundamentals behind libraries tests compiled via crossgen2 #75230

Enable the fundamentals behind libraries tests compiled via crossgen2 #75230

davidwrighton commented Sep 8, 2022

dotnet-issue-labeler bot commented Sep 8, 2022

ghost commented Sep 8, 2022

davidwrighton Sep 8, 2022

MichalStrehovsky Sep 8, 2022

davidwrighton Sep 8, 2022

MichalStrehovsky Sep 8, 2022

davidwrighton Sep 8, 2022

ivdiazsa Sep 8, 2022 •

edited

Loading

ivdiazsa left a comment

trylek left a comment

MichalStrehovsky Sep 9, 2022

ViktorHofer Sep 12, 2022

ViktorHofer Sep 12, 2022

ViktorHofer Sep 12, 2022

ViktorHofer Sep 12, 2022 •

edited

Loading

ivdiazsa commented Oct 13, 2022

davidwrighton Dec 7, 2022

ivdiazsa commented Jan 20, 2023

		Each test project can potentially have multiple target frameworks. There are some tests that might be OS-specific, or might be testing an API that is available only on some target frameworks, so the `TargetFrameworks` property specifies the valid target frameworks.

		### Running tests in custom compilation modes


		To run a test in specific mode, simply build the tests after building the prerequisite subsets, and specify the test mode on a command line such as:
		```cmd

	dotnet build /t:Test /p:Configuration=Release /p:TestReadyToRun=true
	dotnet build /t:Test -c Release /p:TestReadyToRun=true

		### Running tests in custom compilation modes

		There are several custom compilation modes for tests. These are enabled by setting a switch during the config.

		<!-- R2R testing does not tolerate the combination of a regular project reference and a content reference. -->
		<PublishReadyToRunExclude Include="LongPath.dll" />

Enable the fundamentals behind libraries tests compiled via crossgen2 #75230

Enable the fundamentals behind libraries tests compiled via crossgen2 #75230

Conversation

davidwrighton commented Sep 8, 2022

dotnet-issue-labeler bot commented Sep 8, 2022

ghost commented Sep 8, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ivdiazsa Sep 8, 2022 • edited Loading

Choose a reason for hiding this comment

ivdiazsa left a comment

Choose a reason for hiding this comment

trylek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ViktorHofer Sep 12, 2022 • edited Loading

Choose a reason for hiding this comment

ivdiazsa commented Oct 13, 2022

Choose a reason for hiding this comment

ivdiazsa commented Jan 20, 2023

ivdiazsa Sep 8, 2022 •

edited

Loading

ViktorHofer Sep 12, 2022 •

edited

Loading