.NET Native Library Packaging (RuntimeIdentifiers, build, testing, VS etc.) #33845
Comments
dotnet-issue-labeler
bot
added
Area-External
untriaged
|
|
This entire space has a large number of issues and there isn't any good or "official" way to do things. Even ClangSharp is doing it the way it is primarily because of NuGet package size limits, but also because no one wants to download a single 256MB or larger package when they only need a 32MB subset of it. Multiple issues, many of which you linked to in the OP, exist that track the general problem space. |
|
Thank you for filling issue. I hope that the volume of these issues will make us to do something about the native dependencies packaging scenario. I agree that the current experience is very poor.
This looks like a bug to me. https://github.com/microsoft/vstest/ would be a better place to discuss this specific issue.
#23540 is the main tracking issue for this change in the default behavior. The change was mentioned in .NET blog posts where you have probably seen it. It addresses the confusing coupling of RID-specific and self-contained that you have touched on. |
|
@mhutch We have discussed the poor experience of using NuGet to distribute native dependencies some time ago. Do you have any updates that you can share? |
|
@jkotas @tannergooding quick question, when using It does not appear to be needed for When I use |
|
If there are multiple native libraries that depend on each other, it is up to them to make that work. Linking these libraries with the correct |
You can use the list of paths from AppDomain.CurrentDomain.GetData(“NATIVE_DLL_SEARCH_DIRECTORIES”) to get the list of directories where native libraries are located. |
|
@jkotas thanks for the replies.
I am not the builder of these libraries, merely the packager. Hence, have no control over linker options. Or library behavior.
I am not directly in charge of loading these libraries and would very much like to avoid it. The situation should be somewhat like the below. Where native libs are in flowchart LR
A[MgdExe] -->|Uses| B[MgdLib]
B -->|P/Invoke| C(NtvLibA)
C -->|LoadLibrary| D[NtvLibB]
Since Hence, I am as far as I can tell I am left with just one option, changing |
|
Actually, |
|
For a published WPF app // Try manually loading the .NET WPF native library dependencies
TryManuallyLoad("vcruntime140_cor3");
TryManuallyLoad("wpfgfx_cor3");
TryManuallyLoad("PresentationNative_cor3");
TryManuallyLoad("D3DCompiler_47_cor3"); |
|
Sorry to spam and bother you guys again, but I have yet another issue that I am scratching my head over. In the application (WPF - .NET 6) where this is to be used, we also have a dependency on an old .NET Fx assembly that is situated in the GAC. We find and load this via fusion i.e. we load var fusionFullPath = Environment.Is64BitProcess
? @"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\fusion.dll"
: @"C:\Windows\Microsoft.NET\Framework\v4.0.30319\fusion.dll";
NativeLibrary.Load(fusionFullPath);then use something like: /// <summary>
/// Gets an assembly path from the GAC given a partial name.
/// </summary>
/// <param name="name">An assembly partial name. May not be null.</param>
/// <returns>
/// The assembly path if found; otherwise null;
/// </returns>
public static string GetAssemblyPath(string name)
{
if (name == null)
{ throw new ArgumentNullException(nameof(name)); }
var hr = CreateAssemblyCache(out var assemblyCache, 0);
if (hr >= 0)
{
var assemblyInfo = new AssemblyInfo();
assemblyInfo.cchBuf = 1024; // should be fine...
assemblyInfo.currentAssemblyPath = new string('\0', assemblyInfo.cchBuf);
hr = assemblyCache.QueryAssemblyInfo(0, name, ref assemblyInfo);
if (hr >= 0)
{
return assemblyInfo.currentAssemblyPath;
}
}
return null;
}
[ComImport, InterfaceType(ComInterfaceType.InterfaceIsIUnknown), Guid("e707dcde-d1cd-11d2-bab9-00c04f8eceae")]
interface IAssemblyCache
{
void Reserved0();
[PreserveSig]
int QueryAssemblyInfo(int flags, [MarshalAs(UnmanagedType.LPWStr)] string assemblyName, ref AssemblyInfo assemblyInfo);
}
[StructLayout(LayoutKind.Sequential)]
struct AssemblyInfo
{
public int cbAssemblyInfo;
public int assemblyFlags;
public long assemblySizeInKB;
[MarshalAs(UnmanagedType.LPWStr)]
public string currentAssemblyPath;
public int cchBuf; // size of path buf.
}
// On .NET 5+ we get the following:
// System.DllNotFoundException: Unable to load DLL 'fusion.dll' or one of its dependencies: The specified module could not be found. (0x8007007E)
// https://github.com/dotnet/core/issues/3048
// To fix this we use NativeLibrary.Load in the static constructor above.
[DllImport("fusion.dll")]
static extern int CreateAssemblyCache(out IAssemblyCache ppAsmCache, int reserved);to find the path of that .NET assembly. We then override: AppDomain.CurrentDomain.AssemblyResolve to handle this. This works fine and has worked without issue. But after using This is all very involved, but such is the real-world of industrial computer vision/AI where we have a lot of dependencies that are often out of our control. External code might be old, might be mixed-mode assemblies and so on. |
|
@jkotas it seems to me that I still have not found a solution above. Problem is calling Using Hence, I am stuck in trying to find a good solution. Using The more I read the more confused I get here. |
That is not correct. Self-contained apps are always RID specific. Portable self-contained apps do not exist. RID specific apps (including self-contained RID specific apps) should have everything in the same directory. They should not be hitting any of the problems with dependencies spread over multiple directories. |
Should I then not be able to modify this first thing in startup (Main) like (where AppDomain.CurrentDomain.SetData("NATIVE_DLL_SEARCH_DIRECTORIES", archDir);and then per Unmanaged (native) library probing is should load from that before anything else? This does not appear to work (for self-contained), it will load from
I understand this and perhaps I did not explain it very well. We need to support BOTH framework-dependent deployment (incl. local debugging in VS) And as I tried to write for self-contained deployment it is a requirement (from us/customers etc.) that the native libraries are located/moved in a sub-directory (e.g. In any case, we naturally must support developers being able run the application from visual studio or whatever with F5 for debugging. With nuget packages following Perhaps to make this more clear I have tried to show the layout for the different scenarios below. Framework-dependent ( Self-contained ( This is of course a demonstrative example. Both of these "layouts" can be used both locally and on production machines. For many different reasons. The framework-dependent scenario cannot be supported with just Note this is all Windows currently, but given I am also trying to ship some of our open source dependencies in nuget packages I am also trying to play nice in the community and publish these in a way that could be used by all. Incl. trying to support the different kinds of deployments/usages so it just works. Hence, I am trying to weigh and understand options here. Note we do not have full control over all our dependencies (one being ONNX runtime for example) and hence not on |
|
https://learn.microsoft.com/en-us/dotnet/core/dependency-loading/loading-unmanaged#pinvoke-load-library-algorithm is a more detailed description of the native library loading algorithm. You should be able to call |
Well that explains it of course 😅 Shouldn't it throw on set then or be documented? https://learn.microsoft.com/en-us/dotnet/core/dependency-loading/default-probing mentions that
Yes I have been and am looking into this. It's just seems incredibly complicated compared to setting: SetDllDirectory(Environment.Is64BitProcess ? @"x64" : @"x86");which is basically what we did before, and which works in concert with "normal search order". I am concerned with ending up with the extremes that TorchSharp had to go for to get a good out-of-the-box experience (something I applaud), see: Having to define this per assembly effectually couples a lot of things together and is harder to reason about and configure for different kinds of use cases. And it doesn't solve transitive library dependencies. Don't want to load things that aren't necessarily needed and so on. In many ways, what appears missing to me is proper OS (Windows) support for adding directories early in search order that do not then completely replace the search order like Just to recap my own thoughts here, there are the following options:
Or any combination of the above. This is harder than it should be. 🙈 |
It is documented in AppDomain.SetData: The cache automatically contains predefined system entries that are inserted when the application domain is created. You cannot insert or modify system entries with this method. A method call that attempts to modify a system entry has no effect; the method does not throw an exception. |
baronfel
added
the
native-assets
…ient() extension method - change the package to reference FoundationDB.Client nuget package - add UseNativeClient() extension method on FdbDatabaseProviderOptions that will locate the correct library on disk - handle both Portable and platform specific build and publish - For now, we include the binaries for all the platforms (~52MB), maybe later find a way to split into multiple libs for each platform? (seems to be not that easy, cf dotnet/sdk/issues/33845)
cc: @tannergooding @richlander @jkotas
This is yet another issue regarding how to best author native library nuget packages and define, build, test, publish deploy applications that consume these. I have tried hard to wrap my head about this by reading many issues and studying existing packages. I have a particular need that is similar to
TorchSharpwith massive native libraries that not only need to be split into fragments but also where if possible it would be best only to "download" the runtime identifier (RID) specific packages needed for local development. (But on windows that local development often means BOTH x86 and x64 in our case).Below I wrote a walk-through I did of using
ClangSharp(in excessive detail for reference) and the many questions that it raised for me compared to how I am used to working with this (based on our own way of authoring native library packages that are explicitly copied to sub-directories (x64,x86) alongsideexeand with those directories then added at runtime based on the process arch/os/system to dll directories i.e. viaAddDllDirectory. Having something "custom" is a maintenance issue of course, but also an on-boarding issue. Using documented best practices would be best, but as far as I can tell there are none?In any case, at the end of the walk-through I encounter the problem that when specifying multiple RIDs i.e.
then the
runtime.jsontrick does not appear to work when running unit tests from inside Visual Studio. I have to explicitly add the RID specific nuget packages anyway, so I then wonder how exactly is one supposed to author nuget packages to be able to support running multiple RIDs (in this case solely interested in win-x86 and win-x64 for now) with full support for it as usual in VS and other tools? We need to be able to debug and run from VS?And how do you switch which RID you run with when F5 running in VS?
Should I simply accept that the
runtime.jsonway is too flawed and explicitly reference all needed nuget packages? Would this then avoid the need to specify RIDs? Which also has issues with "forcing" self-contained (we don't want that), in fact we'd like to simply be able to deploy/copy-paste build output as something like:where the app is not RID specific (framework-dependent of course). And this should work on both win-x86/winx64. This is what we have now and what works. Our developers are used to this. But it's based on native library nuget packages that explicitly copy their native library contents to those folders and of course referencing all those RID specific ones. I had hoped perhaps one could avoid the RID specific referencing, but that does not seem to work "smoothly". Which I'd guess then means the whole
runtime.jsonis not the way to go.Secondly, I think I read somewhere (can't find or remember where) that for .NET 8 it is considered to force a specific RID on build? I can see given my experience below why one might consider doing that, but that would then raise other issues such as losing what used to be a core tenant (IMHO) of .NET which is that a build output (not publish) is RID agnostic. Would that be lost then?
All in all, to solve these issues I have to author my own little tool for packaging the native libraries, consider all the issues around consumption, testing etc. And after going through all this I am still left with feeling rather lost 😅 I still don't know exactly what is the best solution here. And the packages I am creating are intended to be published for the public, e.g. so I can publish the revived CNTK packages I've made on nuget.org for example.
On top of this we still want to support publishing RID specific applications, but then we don't want native libraries embedded in single file, there is an option for that which is great, but then we want those dlls in sub-folder, not directly next to the
exe, which means we have to hack around that in MSBuild and then face issues with mixed-mode assemblies etc. Yes, we also have those which also makes things very interesting.ML/AI isn't going away. For each new CUDA or whatever release the native libraries double in size (minimum!). Easy authoring and consumption of those would be great, but I am sure also won't be solved in the immediate future, I need to know what to do now?
The walk-through will come as the next comment.
Links
[Feature] Increase the package size limit on NuGet.org from 250 MB NuGet/NuGetGallery#9473
This features a discussion on how to split a nuget package and links.
https://www.nuget.org/packages/libtorch-cuda-11.7-win-x64/
https://www.nuget.org/packages/SciSharp.TensorFlow.Redist-Linux-GPU#dependencies-body-tab
runtime.jsonhttps://natemcmaster.com/blog/2016/05/19/nuget3-rid-graph/
Improve handling of native packages (Support RID specific dependencies) NuGet/Home#10571
https://www.nuget.org/packages/libclang
https://www.nuget.org/packages/libclang.runtime.win-x64/
https://learn.microsoft.com/en-us/visualstudio/msbuild/msbuild-roslyncodetaskfactory?view=vs-2022
https://natemcmaster.com/blog/2017/07/05/msbuild-task-in-nuget/
Architecture-specific folders like
runtimes/<rid>/native/outside of NuGet packages [nativeinterop] #24708Should
runtime.packages be listed in NuGet.org? core#7568Create a nuget package vincenzoml/SimpleITK-dotnet-quickstart#1
Add a way to list native assets that a project will load from the app directory (list them in deps.json) #11373
Guide for packaging C# library using P/Invoke to per-architecture and/or per-platform C++ native DLLs NuGet/Home#8623
dotnet-nativetemplate git repositoryhttps://github.com/Mizux/dotnet-native
The text was updated successfully, but these errors were encountered: