Testmerging JIT/Directed

[BrianBohe]

This another pr merging JIT tests as PR80597. It starts with these commits that prepare the directory to use later ILTransform like:

• Removing a C# test with its proj file
• Removing csproj that were cloned with just capitalized values
• Refactoring some C# tests on JIT/Directed/nullabletypes/: they were including a file, invokecommon.cs, which triggers the tests. These tests break with the new merged tests as they share invokecommon main class. It seems that it is only important to have different class names for the classes that have a Fact attribute.
• The next 3 commits (1, 2 and 3) deal with main test methods arguments that are used in the test. They are not set on project files and probably were being used locally. I left one of them to run locally.
• Removing double namespace uses
• Fixing TestLibrary references
• Renaming a test to avoid problems later when deduplicating with ILTransform
• Replacing two csproj with one using a condition
• Renaming main test method in order to be found by ILTransform later

Then cs-main and ILTransform projects are used with the arguments specified on commits in the directory. This is the version of the tool I used. Some observations:

• "ILTransform -n" is an heuristic approach to deduplicate project files, so it may need to run more than once to actually deduplicate project files, in this case 3 times.
• Tester test on debuginfo does not need a transformation so this commit changes it back.
• ILTransform -public is too naive, so it may add duplicated "public" tokens when they are, this commit removes them. It could be that private methods, that could still be private, are changed to public exposing private types, this commit fix these cases. It seems it also missed a few cases added later on this commit.

[ghost]

Tagging subscribers to this area: @dotnet/area-system-reflection-metadata
See info in area-owners.md if you want to be subscribed.

Issue Details

---

This another pr merging JIT tests as PR80597. It starts with these commits that prepare the directory to use later ILTransform like:

• Removing a C# test with its proj file
• Removing csproj that were cloned with just capitalized values
• Refactoring some C# tests on JIT/Directed/nullabletypes/: they were including a file, invokecommon.cs, which triggers the tests. These tests break with the new merged tests as they share invokecommon main class. It seems that it is only important to have different class names for the classes that have a Fact attribute.
• The next 3 commits (ce3d4c, 17bebe and 8ff5b0) deal with main test methods arguments that are used in the test. They are not set on project files and probably were being used locally. I left one of them to run locally.
• Removing double namespace uses
• Fixing TestLibrary references
• Renaming a test to avoid problems later when deduplicating with ILTransform
• Renaming main test method in order to be found by ILTransform later

Then cs-main and ILTransform projects are used with the arguments specified on commits in the directory. This is the version of the tool I used. Some observations:

• "ILTransform -n" is an heuristic approach to deduplicate project files, so it may need to run more than once to actually deduplicate project files, in this case 3 times.
• Tester test on debuginfo does not need a transformation so 8c964b changes it back.
• ILTransform -public is too naive, so it may add duplicated "public" tokens when they are, this commit removes them. It could be that private methods, that could still be private, are changed to public exposing private types, this commit fix these cases. It seems it also missed a few cases added later on this commit .

Author:	BrianBohe
Assignees:	BrianBohe
Labels:	area-System.Reflection.Metadata
Milestone:	-

[BrianBohe]

Before adding the wrapper, the test run looks is:

Time [secs] | Total | Passed | Failed | Skipped | Assembly Execution Summary
============================================================================
    311.617 |   648 |    648 |      0 |       0 | JIT.Directed.XUnitWrapper.dll
----------------------------------------------------------------------------
    311.617 |   648 |    648 |      0 |       0 | (total)

this can be get running git checkout .; git clean -xdf; ./build clr+libs -rc debug -lc Release;.\src\tests\build debug generatelayoutonly; .\src\tests\build tree JIT\Directed -priority=1 rebuild; .\src\tests\run; on this branch

With the last commit on this PR, the tests run look like this:

Time [secs] | Total | Passed | Failed | Skipped | Assembly Execution Summary
============================================================================
     86.431 |    48 |     48 |      0 |       0 | JIT.Directed.Directed_cs_r
     70.659 |    48 |     48 |      0 |       0 | JIT.Directed.Directed_cs_ro
    118.592 |   131 |    131 |      0 |       0 | JIT.Directed.Directed_d
     99.982 |    73 |     73 |      0 |       0 | JIT.Directed.Directed_do
    163.975 |   216 |    215 |      0 |       1 | JIT.Directed.Directed_others
     39.468 |    94 |     94 |      0 |       0 | JIT.Directed.Directed_r
     36.185 |    36 |     36 |      0 |       0 | JIT.Directed.Directed_ro
      0.477 |     0 |      0 |      0 |       0 | JIT.Directed.Directed_unix
      7.562 |     1 |      1 |      0 |       0 | JIT.Directed.Directed_windows
----------------------------------------------------------------------------
    623.332 |   647 |    646 |      0 |       1 | (total)

There are multiple failures on this last output:

1. Test JIT\Directed\rvastatics\RVAOrderingTest.ilproj is skipped on Directed_others now with no apparent reason (<reason><![CDATA[No Known Skip Reason]]></reason>) and run fine before merging tests.
2. It seems I am missing 2 tests but the test run has a diff of 1, which I found really confusing. Two tests are being built but not run now (and run before merging tests):

• JIT\Directed\IL\leave\leave1.ilproj (and ex leave2.il now leave1.il)
• JIT\Directed\Misc\function_pointer\MutualThdRecur-fptr.ilproj (il file): which actually should be skipped as it is added twice here and here

3. Time of the test run is twice as before. 1/5 of the total amount of tests need to be executed isolated but there should be an improvement. (Although the table summary displays a total run of ~610 seconds, I am clocking it and run in less than ~170 seconds. The table might be adding each test group time as if they run sequentially. Time is no longer a problem.)

[BrianBohe]

Related issue 71732

[BrianBohe]

There are errors building tests on different non-windows platforms but I cannot reproduce them. I tried emulating the same lines the pipeline is executing in a docker container following these steps:

1. running docker on wsl2: docker run -it -v /<path_to_runtime>:/runtime -w /runtime mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7 (this is the image used in pipeline "runtime (Build coreclr Common Pri0 Test Build AnyOS AnyCPU checked)")
2. Launch bash inside the docker console session and run: ./build.sh libs.sfx+libs.oob+clr.iltools /p:RefOnly=true -c Release -ci --clang9 (command taken from logs on build coreclr step)
3. Then build the tests running: ./src/tests/build.sh -log:Managed allTargets skipnative skipgeneratelayout skiptestwrappers checked x64 ci /p:TargetOS=AnyOS (command taken from logs on build managed test components)

That is not failing, and tests can be run then with ./src/tests/run.sh

[trylek]

@BrianBohe - I assume you're talking about the "entry point is global code" error. Unfortunately @jkoritzinsky is OOF until next Tuesday, I think he would be best suited to comment on that, I guess it's some shenanigans around Roslyn language version - as I see it, the global entrypoint is the Roslyn-generated wrapper, not some hand-written code. Having said that, it's quite weird that it only affects just a handful of tests, if the simple test wrappers generating global entry points are the real problem, it should be affecting 3/4 of our tests and I don't see that in the log.

[markples]

nullable.cs now has a [Fact] Main method, but it should have a different name. Note also that I think you can undo the change of moving the test entry out of invokecommon.cs. I'm guessing that this was done to dedup the entry point's namespace, but it is no longer necessary to do that.

[BrianBohe]

This is how the test run summary looks like now (the host machine is different, so times are different):

Time [secs] | Total | Passed | Failed | Skipped | Assembly Execution Summary
============================================================================
    219.418 |    48 |     48 |      0 |       0 | JIT.Directed.Directed_cs_r
    187.806 |    48 |     48 |      0 |       0 | JIT.Directed.Directed_cs_ro
    262.771 |   131 |    131 |      0 |       0 | JIT.Directed.Directed_d
    236.093 |    73 |     73 |      0 |       0 | JIT.Directed.Directed_do
    293.788 |   127 |    126 |      0 |       1 | JIT.Directed.Directed_others_group1
     86.407 |    92 |     92 |      0 |       0 | JIT.Directed.Directed_others_group2
    102.000 |    94 |     94 |      0 |       0 | JIT.Directed.Directed_r
     92.298 |    36 |     36 |      0 |       0 | JIT.Directed.Directed_ro
----------------------------------------------------------------------------
   1480.582 |   649 |    648 |      0 |       1 | (total)

Tests Directed\IL\leave\leave2.il and Directed\Misc\function_pointer\MutualThdRecur-fptr.il were fixed wrapping the entry point, which was a "global static public int32 main", with a "public class".

The skipped test is Directed\rvastatics\RVAOrderingTest.ilproj.

[BrianBohe]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[markples]

RVAOrderingTest appears to be excluded in issues.targets:

        <ExcludeList Include="$(XunitTestBinBase)/JIT/Directed/rvastatics/RVAOrderingTest/*">
            <Issue>https://github.com/dotnet/runtime/issues/55966</Issue>
        </ExcludeList>

I would expect it to be skipped in the current runs as well. Are you not seeing that?

[BrianBohe]

There are many tests that ended up running and passing after being renamed, I have updated those paths on issues.targets but may be a good moment to unable them back. Here is one of the targeted issues

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

---

This another pr merging JIT tests as PR80597. It starts with these commits that prepare the directory to use later ILTransform like:

• Removing a C# test with its proj file
• Removing csproj that were cloned with just capitalized values
• Refactoring some C# tests on JIT/Directed/nullabletypes/: they were including a file, invokecommon.cs, which triggers the tests. These tests break with the new merged tests as they share invokecommon main class. It seems that it is only important to have different class names for the classes that have a Fact attribute.
• The next 3 commits (1, 2 and 3) deal with main test methods arguments that are used in the test. They are not set on project files and probably were being used locally. I left one of them to run locally.
• Removing double namespace uses
• Fixing TestLibrary references
• Renaming a test to avoid problems later when deduplicating with ILTransform
• Replacing two csproj with one using a condition
• Renaming main test method in order to be found by ILTransform later

Then cs-main and ILTransform projects are used with the arguments specified on commits in the directory. This is the version of the tool I used. Some observations:

• "ILTransform -n" is an heuristic approach to deduplicate project files, so it may need to run more than once to actually deduplicate project files, in this case 3 times.
• Tester test on debuginfo does not need a transformation so this commit changes it back.
• ILTransform -public is too naive, so it may add duplicated "public" tokens when they are, this commit removes them. It could be that private methods, that could still be private, are changed to public exposing private types, this commit fix these cases. It seems it also missed a few cases added later on this commit.

Author:	BrianBohe
Assignees:	BrianBohe
Labels:	area-CodeGen-coreclr
Milestone:	-

[BrianBohe]

I checked the whole file change and seems to be ok but I am not sure about losing this. Do you know @jakobbotsch ?

[BrianBohe]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[BrianBohe]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[BrianBohe]

@trylek
I can see on logs that most of failures looks like (example):
Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'C:\h\w\AE57097E\w\AD9C0917\e\JIT\Directed\Directed_others_group2\Directed_others_group2.dll'. The system cannot find the file specified.
Have you seen something like that before with Methodical?

Also, in step Generate test wrappers of coreclr Pri0 Runtime Tests Run windows x64 checked I can see that JIT.Directed.XUnitWrapper.dll is still being built in place of the Directed_r/ro/d/*.dlls . Could it be related to the way we are referencing the test groups?

[trylek]

Thanks Brian for the heads-up. I guess I'll need to play with this for a bit locally to see what's gone awry. For generation of the legacy wrapper, that typically signifies that not all tests have been switched over - basically, the legacy XUnit wrapper tests are generated based on the previously generated cmd / sh test execution scripts, the MSBuild script just collects them and passes them to the XUnit wrapper generator (unless they're specifically marked as out-of-process items of a merged test wrapper).

[markples]

Other failures were

• I suggested using DefineConstants in C# to combine a Windows/Linux test, but we used an AnyOS C# build
• Missing updates in issues.targets for project renames

This PR will be replaced by #83256 so that I can write to it.