Using .NET Core COM server from new .NET Core Single-File app results in two runtimes in process

[vitek-karas]

Client:
.NET 5 console app which uses a COM object
Published as new single-file self-contained

Server:
.NET 5 COM Server implementing the COM object

Using reg-free COM, but this should not really matter.

The code actually works, but it loads two coreclr.dll modules into the process. The first coreclr comes from the client app (it's the coreclr.dll which is next to the bundle). The second coreclr is loaded from the global install.

The underlying issue is that comhost looks for hostfxr in the process, but in case of single-file it doesn't find it, so it assumes there's no .NET Core runtime in the process yet, loads hostfxr from global install and continues on by loading the runtime from there as well.

It works correctly when the client is no a single-file app, or if it's an FDD single-file app. So the only broken case is single-file SCD app.

[ghost]

Tagging subscribers to this area: @swaroop-sridhar, @agocke
See info in area-owners.md if you want to be subscribed.

[vitek-karas]

This is not new - we've realized this problem pretty early on (unfortunately sort-of forgot since). Writeup of the problem I did some time ago:

Native component hosting with single-file bundles

Native component hosting is a set of functionality which allows native code to dynamically load managed code into a process. If the process is .NET Core application, this allows external callers to load managed code into the existing runtime in the process. This is currently done by finding the hostfxr module in the process and calling its exports. With single-exe self-contained bundle, there is no hostfxr module anymore. So the existing solution doesn't work anymore.

We will need to find a new solution and also consider providing some way to solve backward compatibility.

TODO - determine the size implications - this may need to include code in the bundle which will not be used by normal app startup (APIs to be called from the native hosting scenarios).

New solution

For now I suggest we wait for the diagnostics team to decide on a solution which the debuggers will use to find and contact the runtime in a process. That is essentially the same problem. If the debugger solution is acceptable for native hosting as well, we should use it. It's easier to maintain and support one way of doing it.

In any case the single-exe bundle will need to include code to support native hosting, so basically the equivalent functionality to existing hostfxr and hostpolicy which handles these cases. That should not be a big amount of code hopefully as it can be specialized for the case where runtime already exists (unlike existing hosting which has to handle also the case where there's no runtime and starting it).

The new solution will also mean that all hosts (comhost, ijwhost, winrthost and nethost) will need to support the new way of finding the runtime in the process. This means new versions of these components.

Unfortunately, the nethost as is will very likely not be possible to patch like that. Currently it exposes a single API which returns a full path to the hostfxr.dll, which the calling code loads through LoadLibrary and gets exports from to call to perform the actual hosting operations. Since single-exe self-contained bundles won't have hostfxr.dll anywhere, there's no path this can return now. It will probably mean designing a new API in nethost which is more general. Could be something which returns an "interface" (struct of function pointers) which exposes the same functionality as the hostfxr exports. For cases where there is a hostfxr this would basically return the hostfxr exports (so instead of returning a path, it would load hostfxr as well), or in the case of single-exe it would somehow find the runtime and return functions from it.

Backward compatibility consideration

No matter what the new solution looks like, it's not going to be compatible with .NET Core 3.0 native hosting support. This means that for example COM object implemented in .NET Core 3.0 will not be able to load into a .NET 5 single-exe self-contained app (even though it would allow it through roll forward policies).

We could either say that such scenario is now not supported for single-exe self-contained apps.

With the new hosts as described above, we could support rebuilding the existing 3.0 components with a new SDK which would include the new hosts with the new functionality. There would be no need to change the components themselves, basically just a forced rebuild.

There's also a possibility of solving this for at least some cases automatically. On machines which would have globally installed hostfxr from .NET 5 (or higher), trying to load for example a COM component into a process will:

• Look for hostfxr module in the process - which it won't find for single-exe bundle
• as a fallback look for hostfxr in global install locations - which it would find.

The new .NET 5 hostfxr could include the logic to find the existing runtime in the process (for single-exe bundles) and correctly implement its functionality (just like it does today for normal apps)
This way even existing unchanged 3.0 components would be able to load into single-exe bundles if the machine has .NET 5 installed globally.
Note that there's a relatively high chance that the machine will have at least .NET Core 3.0 installed, since COM object registration is not possible without it (COM object can't be self-contained), so we might be able to solve this by shipping an update to 3.0/3.1 hostfxr.

This solution would also us to provide an easy workaround for users - if they run into the problem of .NET 5 single-exe not able to load some 3.0 component, just install .NET 5 runtime on the machine and it should work (since that will put .NET 5 hostfxr on the machine).

Such solution is obviously not perfect, but it's the best one I can think of without changing already shipped 3.0 code.

At the very least we should try to block this actively in 3.1, so that 3.1 COM objects get a consistent failure behavior. See dotnet/core-setup#8299 for more details.

[vitek-karas]

See also discussion in #3773: The outcome seems to be that it's OK to not actively block this scenario if we document it as unsupported (we don't support two runtimes in one process).

[vitek-karas]

/cc @elinor-fung @AaronRobinsonMSFT @agocke @VSadov