Handle open types to appear in interface maps

[davidwrighton]

Fix the following scenarios

• Reflection IsAssignableFrom api
• As well as constraint checking

Fixes #97667

[lambdageek]

COM interop also has a call to IsSpecialMarkerTypeForGenericCasting

runtime/src/coreclr/vm/classcompat.cpp

Lines 1244 to 1245 in 7459797

	if (pItf->HasInstantiation() || pItf->IsSpecialMarkerTypeForGenericCasting())
	continue;

COM can't have the curiously recurring generic scenario, though, right?
Should we assert anyway?

[davidwrighton]

COM interop also has a call to IsSpecialMarkerTypeForGenericCasting

runtime/src/coreclr/vm/classcompat.cpp

Lines 1244 to 1245 in 7459797

	if (pItf->HasInstantiation() || pItf->IsSpecialMarkerTypeForGenericCasting())
	continue;

COM can't have the curiously recurring generic scenario, though, right? Should we assert anyway?

Unfortunately, we can't effectively assert there. We might or might not have the flag set, and in either case it doesn't really make any sense in the presence of COM interop.

[lambdageek]

@davidwrighton Can we do a call Thursday or Friday to review. I'm having some trouble following what's going on (new codebase + curiously recursive generics == 🤯 )

[davidwrighton]

@lambdageek, absolutely, just schedule drop something onto my schedule

[lambdageek]

I also can't make the example convoluted enough to hit the variant case without making the bug go away (because I end up passing at least one generic type argument that is not just a generic parameter - so we don't get the typical instantiation).

The code as written matches what we thought it ought to do during the code review 😄

[davidwrighton]

/backport to release/8.0-staging

[github-actions]

Started backporting to release/8.0-staging: https://github.com/dotnet/runtime/actions/runs/7834567349

[github-actions]

@davidwrighton backporting to release/8.0-staging failed, the patch most likely resulted in conflicts:

$ git am --3way --ignore-whitespace --keep-non-patch changes.patch

Applying: Handle open types to appear in interface maps - Reflection IsAssignableFrom api - As well as constraint checking
Using index info to reconstruct a base tree...
M	src/coreclr/vm/methodtable.cpp
M	src/coreclr/vm/methodtable.h
M	src/coreclr/vm/methodtable.inl
M	src/coreclr/vm/methodtablebuilder.cpp
M	src/tests/Loader/classloader/generics/Instantiation/Positive/CuriouslyRecurringThroughInterface.cs
Falling back to patching base and 3-way merge...
Auto-merging src/tests/Loader/classloader/generics/Instantiation/Positive/CuriouslyRecurringThroughInterface.cs
CONFLICT (content): Merge conflict in src/tests/Loader/classloader/generics/Instantiation/Positive/CuriouslyRecurringThroughInterface.cs
Auto-merging src/coreclr/vm/methodtablebuilder.cpp
Auto-merging src/coreclr/vm/methodtable.inl
Auto-merging src/coreclr/vm/methodtable.h
CONFLICT (content): Merge conflict in src/coreclr/vm/methodtable.h
Auto-merging src/coreclr/vm/methodtable.cpp
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Handle open types to appear in interface maps - Reflection IsAssignableFrom api - As well as constraint checking
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Error: The process '/usr/bin/git' failed with exit code 128

Please backport manually!

[github-actions]

@davidwrighton an error occurred while backporting to release/8.0-staging, please check the run log for details!

Error: git am failed, most likely due to a merge conflict.