Installer test BundleProbe.Bundle_Probe_Passed_For_Single_File_App failures on Linux_musl x64

[safern]

Builds

Build	Pull Request	Test Failure Count
#626420	Rolling	1
#626513	Rolling	1
#626634	Rolling	1
#626761	Rolling	1
#626950	Rolling	1
#627020	Rolling	1
#627303	Rolling	1
#627435	Rolling	1
#627770	Rolling	1
#627900	Rolling	1
#627940	Rolling	1
#627985	Rolling	1
#627992	Rolling	1
#628150	Rolling	1
#628240	Rolling	1

Configurations

• Installer-coreclr-Linux_musl_x64-Release

Expected command to pass but it did not."\nFile Name: /root/runtime/artifacts/tests/Release/ahb/1/BundleProbeTester/bundle/BundleProbeTester\nArguments: SingleFile\nExit Code: 253\nStdOut:\nUnexpected file hostpolicy.dll found in bundle.\n\nStdErr:\n\n"

   at FluentAssertions.Execution.XUnit2TestFramework.Throw(String message)
   at FluentAssertions.Execution.TestFrameworkProvider.Throw(String message)
   at FluentAssertions.Execution.DefaultAssertionStrategy.HandleFailure(String message)
   at FluentAssertions.Execution.AssertionScope.FailWith(String message, Object[] args)
   at Microsoft.DotNet.CoreSetup.Test.CommandResultAssertions.Pass() in /_/src/installer/test/TestUtils/Assertions/CommandResultAssertions.cs:line 31
   at AppHost.Bundle.Tests.BundleProbe.Bundle_Probe_Passed_For_Single_File_App() in /_/src/installer/test/Microsoft.NET.HostModel.Tests/AppHost.Bundle.Tests/BundleProbe.cs:line 46

https://dev.azure.com/dnceng/public/_build/results?buildId=628240&view=ms.vss-test-web.build-test-results-tab&runId=19618302&resultId=100000&paneView=debug

Docker image: mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.9-WithNode-0fc54a3-20190918214015

@swaroop-sridhar this is blocking clean rolling builds, can you please take a look?

cc: @dotnet/runtime-infrastructure

[ghost]

Tagging subscribers to this area: @ViktorHofer
Notify danmosemsft if you want to be subscribed.

[ghost]

Tagging subscribers to this area: @swaroop-sridhar
Notify danmosemsft if you want to be subscribed.

[swaroop-sridhar]

OK I'll take a look into it.
I'll disable the test in the meantime so that rolling builds are green.

[safern]

Thanks, @swaroop-sridhar

[safern]

Tests where disabled so removing the blocking-ci label.

[swaroop-sridhar]

we're hitting a set of test failures only on Linux-Musl release jobs.
This issue was the first test to fail so we've temporarily disabled it.

I've reproed the problem locally using the docker image. It looks like a misinterpreted return value (maybe due to compiler optimization) at an interop boundary for stdcall. So, @AaronRobinsonMSFT I wanted to ask your opinion about how to debug it.

In this test, the program PInvokes to an Stdcall function. This calls bundle_probe function in the host.

For the invocations that test for non-existant files, the call eventually returns false from native code, but the managed code thinks that it got a true, and fails at this step.

So, was wondering if this rings a bell.

There are several more here: #35967
Not sure if they are the same problem ...

CC: @VSadov @vitek-karas

[AaronRobinsonMSFT]

So the StdCall should not impact this at all - that should only impact x86. However that is an interesting place to start. What would be helpful is to see the generated IL stub for the call.

Set a breakpoint at line 1103.

runtime/src/coreclr/src/vm/dllimport.cpp

Lines 1103 to 1115 in 08b244e

	if (ETW_EVENT_ENABLED(MICROSOFT_WINDOWS_DOTNETRUNTIME_PROVIDER_DOTNET_Context, ILStubGenerated))
	{
	EtwOnILStubGenerated(
	pStubMD,
	pbLocalSig,
	cbSig,
	jitFlags,
	&tryCatchClause,
	&cleanupTryFinally,
	maxStack,
	(DWORD)cbCode
	);
	}

The predicate will evaluate false, so you need to set the IP manually to be on 1105.

Set a breakpoint at line 1255.

runtime/src/coreclr/src/vm/dllimport.cpp

Lines 1252 to 1267 in 08b244e

	//
	// Fire ETW event
	//
	FireEtwILStubGenerated(
	GetClrInstanceId(), // ClrInstanceId
	uModuleId, // ModuleIdentifier
	(UINT64)pStubMD, // StubMethodIdentifier
	dwFlags, // StubFlags
	dwToken, // ManagedInteropMethodToken
	strNamespaceOrClassName.GetUnicode(), // ManagedInteropMethodNamespace
	strMethodName.GetUnicode(), // ManagedInteropMethodName
	strMethodSignature.GetUnicode(), // ManagedInteropMethodSignature
	strNativeSignature.GetUnicode(), // NativeSignature
	stubMethodSignature.GetUnicode(), // StubMethodSigature
	strILStubCode.GetUnicode() // StubMethodILCode
	);

From here you can look at strILStubCode and that will contain the generated IL stub.

[AaronRobinsonMSFT]

It would also be helpful to see the native function being called - at least the signature.

[swaroop-sridhar]

@AaronRobinsonMSFT The native function being called is bundle_probe with signature
bool (anonymous namespace)::bundle_probe(char16_t const*, long*, long*)

Since this is a release build only failure, I'm having a hard time setting a breakpoint at the said location, I'll try a source change to make it work -- and let you know. Thanks.