Improve handling of native packages (Support RID specific dependencies)

[tannergooding]

Issue

Native dependencies tend to be OS/architecture specific and larger than managed assemblies. Due to this, it isn't uncommon to end up with a NuGet package that is "unreasonably" large due to needing to support at least the 4 most common architectures (x86 Windows, x64 Windows, x64 Linux, x64 MacOS). This only increases once you also factor in Linux Distro specific variants that are often required (Ubuntu vs RHEL) or up and coming architectures (ARM32/64).

Native dependencies such as libClang are currently ~22MB per RID (compressed) and so they are using the runtime.json functionality to have a small central dependency (libclang, 16.75kb) which refers to the larger RID specific dependencies of which there are currently 11. For example: ((libclang.runtime.win-x64, 22.33mb). A similar situation exists for libLLVM.

runtime.json works very well at limiting the usage to only what the user actually needs. However, the experience is less than stellar when not specifying a RID as the downstream native dependencies aren't copied to the output folder and are therefore not discoverable at runtime. And, at least as I understand, runtime.json is not "officially" supported for general usage

• This differs in how directly referencing a RID specific package (e.g. directly referencing libclang.runtime.win-x64 vs referencing libclang) works in that the relevant binaries do get copied.

Workaround

The current workaround for this has been to explicitly specify a RID at build time, such as:

<PropertyGroup>
    <RuntimeIdentifier Condition="'$(RuntimeIdentifier)' == ''">$(NETCoreSdkRuntimeIdentifier)</RuntimeIdentifier>
</PropertyGroup>

However, this means that users need to go through an additional step beyond simply adding their dependency and need to understand what the consequences of specifying a RID are. Additionally, NETCoreSdkRuntimeIdentifier may be linux-x64 in which case it won't resolve to the actual RID specific binary that is desired (such as ubuntu.20.04-x64).

Requested Fix

It would be ideal if there was an official way to have RID specific native dependencies that could be easily and correctly handled from .NET libraries.

Users should be able to reference a central "meta" package that lists the RID specific dependencies so they download the minimum binaries required, particularly when in a development environment.

The correct native dependencies should be resolvable under F5 by DllImport and NativeLibrary.Load without the user needing to specify additional project configuration settings.

There should be some mechanism for resolving native dependencies when things don't line up. That is, you should be able to use ubuntu18.04 from ubuntu20.04 if not 20.04 specific dependency does exist. It would (IMO) be fine if this was via explicit opt-in at the individual package level or if it was enabled by default with a warning.

[FiniteReality]

It'd be nice if NuGet defaulted to the RID of the host system without the user having to specify RuntimeIdentifier manually, so that it could download the correct packages for the host system in development environments, and then only fail when attempting to publish.

[AntonLapounov]

It would be nice to allow package dependencies conditioned on the host RID. For instance, ILCompiler supports targeting win-arm64 RID on a win-x64 host using the dotnet publish -r win-arm64 command. To make that work, the ILCompiler meta-package should list a dependency on the win-x64 tools package, containing the win-x64-hosted compiler, and the win-arm64 runtime package, containing the framework for the app.

[tannergooding]

dotnet/ClangSharp#229 is another example of where native package handling could be improved.

When a RID specific package contains ubuntu.18.04-x64 but not linux-x64, then the dependency is not copied because the SDK sets the RID to linux-x64 (even when the host is actually ubuntu.18.04-x64). Manually coding the RID resolution rules into an app is essentially a non-starter and it would be good for NuGet to do the right thing here.

[FiniteReality]

Another issue I've discovered recently is that GeneratePathProperty for a package which uses a runtime.json isn't correctly updated to point to the path of the package specified in the runtime.json - this makes using artefacts from packages like https://www.nuget.org/packages/Microsoft.NETCore.DotNetAppHost when hosting .NET Core a lot harder.

[nkolev92]

Hey @tannergooding
Do you think this is similar to #1660?

It seems like we didn't dedup it originally, but there seems to be a lot of overlap.