Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Most CoreCLR tests not running in CI. #35798

Closed
sdmaclea opened this issue May 4, 2020 · 8 comments
Closed

Most CoreCLR tests not running in CI. #35798

sdmaclea opened this issue May 4, 2020 · 8 comments
Assignees
@sdmaclea
Labels
area-Infrastructure-coreclr
Milestone
5.0.0

Comments

@sdmaclea
Copy link
Contributor

sdmaclea commented May 4, 2020

There are several issues currently.

The most significant is that the targetGeneric tests are not currently running in CI.

The root cause of this is the unzipping of the targetSpecific and targetGeneric tests into the same directory is broken, because the second download clean the directory removing the targetGeneric tests. The draft fix is #35699.

It also turns out that #35645 fix to eliminate the targetSpecific build legs would also fix the directory cleaning issue.

However #35699 & #35645 both demonstrates another set of issues.

  • The generic tests include Linux specific tests. When run on Windows these fail.
  • The specific tests are missing WIndows specific tests.

So in general

  • any test which doesn't run on OSX is not being built in the targetGeneric phase.
  • any test which doesn't run on every target cannot be treated as targetGeneric

The current proposed solution is #35783. This test allows tests for all targets to be build on any platform. It builds all tests on OSX and removes obsolete build legs and cleans up residue.

/cc @dotnet/runtime-infrastructure
@sdmaclea sdmaclea added this to the 5.0 milestone May 4, 2020
@sdmaclea sdmaclea self-assigned this May 4, 2020
@Dotnet-GitSync-Bot Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label May 4, 2020
@sdmaclea sdmaclea changed the title All CoreCLR tests not running in CI. Most CoreCLR tests not running in CI. May 4, 2020
@sdmaclea sdmaclea mentioned this issue May 4, 2020
Closed
@jashook jashook removed the untriaged New issue has not been triaged by the area owner label May 4, 2020
@sdmaclea
Copy link
Contributor Author

sdmaclea commented May 6, 2020 •
edited
Loading

Coverage gap is being fixed by #35868

@sdmaclea
Copy link
Contributor Author

sdmaclea commented May 6, 2020 •
edited
Loading

The following tests regressed during the coverage gap.

commit b653f7942c415328bf2a22ab61be58da94794b90
Author: Steve MacLean <Steve.MacLean@Microsoft.com>
Date:   Tue May 5 17:12:44 2020 -0400

    Add issues related to CI coverage outage

diff --git a/src/coreclr/tests/issues.targets b/src/coreclr/tests/issues.targets
index ce48665..d57765f 100644
--- a/src/coreclr/tests/issues.targets
+++ b/src/coreclr/tests/issues.targets
@@ -11,6 +11,25 @@
         <ExcludeList Include="$(XunitTestBinBase)/Regressions/coreclr/GitHub_22888/test22888/*">
             <Issue>https://github.com/dotnet/runtime/issues/13703</Issue>
         </ExcludeList>
+        <!-- Test regressions during CI coverage outage -->
+        <ExcludeList Include="$(XunitTestBinBase)/baseservices/exceptions/stackoverflow/stackoverflowtester/*">
+            <Issue>https://github.com/dotnet/runtime/issues/35798</Issue>
+        </ExcludeList>
+        <ExcludeList Include="$(XunitTestBinBase)/JIT/SIMD/Vector3Interop_r/*">
+            <Issue>https://github.com/dotnet/runtime/issues/35798</Issue>
+        </ExcludeList>
+        <ExcludeList Include="$(XunitTestBinBase)/JIT/SIMD/Vector3Interop_ro/*">
+            <Issue>https://github.com/dotnet/runtime/issues/35798</Issue>
+        </ExcludeList>
+        <ExcludeList Include="$(XunitTestBinBase)/Interop/StructMarshalling/ReversePInvoke/MarshalSeqStruct/ReversePInvoke/ReversePInvokeTest/*">
+            <Issue>https://github.com/dotnet/runtime/issues/35798</Issue>
+        </ExcludeList>
+        <ExcludeList Include="$(XunitTestBinBase)/Interop/PInvoke/Vector2_3_4/Vector2_3_4/*">
+            <Issue>https://github.com/dotnet/runtime/issues/35798</Issue>
+        </ExcludeList>
+        <ExcludeList Include="$(XunitTestBinBase)/Interop/UnmanagedCallersOnly/UnmanagedCallersOnlyTest/*">
+            <Issue>https://github.com/dotnet/runtime/issues/35798</Issue>
+        </ExcludeList>
     </ItemGroup>
 
     <!-- All OS/Arch CoreCLR excludes -->

@sdmaclea
Copy link
Contributor Author

sdmaclea commented May 6, 2020

The regressions were found while testing 96268f3
The CI coverage was introduced by #35378 so untested coverage began around commit d56f849

@sdmaclea
Copy link
Contributor Author

sdmaclea commented May 6, 2020

/cc @janvorli for the regressing stackoverflowtester test

/cc @BruceForstall @tannergooding @echesakovMSFT
for the regressing SIMD/Vector3Interop* tests

/cc @AaronRobinsonMSFT for the regressing InterOp tests.

@jashook
Copy link
Contributor

jashook commented May 6, 2020

Closing, we are now back to 29 workitems.

[
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/HelixController%20Work%20Queueing",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "HelixController Work Queueing",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/Interop",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "Interop",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/JIT",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "JIT",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/JIT.CodeGenBringUpTests",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "JIT.CodeGenBringUpTests",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/JIT.Directed",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "JIT.Directed",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/JIT.Generics",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "JIT.Generics",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/JIT.HardwareIntrinsics",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "JIT.HardwareIntrinsics",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/JIT.IL_Conformance",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "JIT.IL_Conformance",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/JIT.Methodical",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "JIT.Methodical",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/JIT.Methodical.a-dA-D",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "JIT.Methodical.a-dA-D",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/JIT.Methodical.e-iE-I",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "JIT.Methodical.e-iE-I",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/JIT.Methodical.eh",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "JIT.Methodical.eh",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/JIT.Regression",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "JIT.Regression",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/JIT.Regression.CLR-x86-JIT.V1-M09-M11",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "JIT.Regression.CLR-x86-JIT.V1-M09-M11",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/JIT.Regression.CLR-x86-JIT.V1-M12-M13",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "JIT.Regression.CLR-x86-JIT.V1-M12-M13",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/JIT.Stress",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "JIT.Stress",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/JIT.jit64",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "JIT.jit64",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/JIT.jit64.hfa",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "JIT.jit64.hfa",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/JIT.jit64.mcc",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "JIT.jit64.mcc",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/JIT.jit64.opt",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "JIT.jit64.opt",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/JIT.jit64.valuetypes",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "JIT.jit64.valuetypes",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/Loader.classloader",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "Loader.classloader",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/Loader.classloader.generics",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "Loader.classloader.generics",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/PayloadGroup0",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "PayloadGroup0",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/baseservices.threading",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "baseservices.threading",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/profiler.eventpipe",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "profiler.eventpipe",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/profiler.gc",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "profiler.gc",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/profiler.rejit",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "profiler.rejit",
        "State": "Finished"
    },
    {
        "DetailsUrl": "https://helix.dot.net/api/2019-06-17/jobs/b3373b29-ba30-42f2-b978-d42b410eb1b1/workitems/profiler.unittest",
        "Job": "b3373b29-ba30-42f2-b978-d42b410eb1b1",
        "Name": "profiler.unittest",
        "State": "Finished"
    }
]

@jashook jashook closed this as completed May 6, 2020
@tannergooding
Copy link
Member

/cc @BruceForstall @tannergooding @echesakovMSFT
for the regressing SIMD/Vector3Interop* tests

This looks to be for @AaronRobinsonMSFT as well. They are all related to Non-blittable parameter types are invalid for UnmanagedCallersOnly methods., which I wouldn't expect for Vector2, Vector3, or Vector4 which should all be blittable and classified as "user-defined" structs for ABI and marshaling purposes.

@tannergooding tannergooding mentioned this issue May 6, 2020
Merged
@AaronRobinsonMSFT
Copy link
Member

@tannergooding The issue is related to returned ValueTypes for delegates marked with UnmanagedCallersOnlyAttributes - which is now done automatically.

runtime/src/coreclr/src/vm/dllimport.cpp

Lines 3506 to 3520 in f4a67df

// return value is fine as long as it can be normalized to an integer
if (i == 0)
{
CorElementType normalizedType = hndArgType.GetInternalCorElementType();
if (normalizedType == ELEMENT_TYPE_VALUETYPE)
{
// it is a structure even after normalization
return TRUE;
}
}
else
{
dwStackSize += StackElemSize(hndArgType.GetSize());
}
break;

I will update the location where we implicitly mark reverse P/Invokes with UnmanagedCallersOnlyAttributes.

runtime/src/coreclr/src/vm/ilstubcache.cpp

Lines 296 to 305 in f4a67df

// mark certain types of stub MDs with random flags so ILStubManager recognizes them
if (SF_IsReverseStub(dwStubFlags))
{
pMD->m_dwExtendedFlags |= DynamicMethodDesc::nomdReverseStub;
#if !defined(TARGET_X86)
pMD->m_dwExtendedFlags |= DynamicMethodDesc::nomdUnmanagedCallersOnlyStub;
#endif
pMD->GetILStubResolver()->SetStubType(ILStubResolver::NativeToCLRInteropStub);
}
else

/cc @jkotas

@AaronRobinsonMSFT
Copy link
Member

I will update the location where we implicitly mark reverse P/Invokes with UnmanagedCallersOnlyAttributes.

Hmm can't do that it seems... I will need to dig more into this.

@AaronRobinsonMSFT AaronRobinsonMSFT mentioned this issue May 6, 2020
Merged
@tannergooding tannergooding mentioned this issue May 7, 2020
Merged
@ghost ghost locked as resolved and limited conversation to collaborators Dec 9, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@sdmaclea sdmaclea

Labels
area-Infrastructure-coreclr
Projects
None yet
Milestone
5.0.0
Development

No branches or pull requests
5 participants
@jashook @tannergooding @sdmaclea @AaronRobinsonMSFT @Dotnet-GitSync-Bot