Native libraries are not resolved

[yuriyostapenko]

Description

When packages referenced by build scripts have RID specific native library dependencies, those assemblies cannot be loaded at runtime.

Repro steps

Please provide the steps required to reproduce the problem

1. dotnet new fake

2. paket.dependencies

// [ FAKE GROUP ]
group Build
    source https://api.nuget.org/v3/index.json
    storage: none
    nuget Fake.Core.Target
    nuget System.Data.SqlClient

3. build.fsx

#load ".fake/build.fsx/intellisense.fsx"
open Fake.Core
open System.Data.SqlClient

Target.create "all" (fun _ ->
    let connectionString = sprintf "Data Source=anything;Initial Catalog=master"
    use connection = new SqlConnection(connectionString)
    connection.Open()
)

Target.runOrDefault "all"

4. fake.cmd build

Expected behavior

Build succeeds (given connection string works, of course)

Actual behavior

DllNotFoundException: Unable to load DLL 'sni.dll' or one of its dependencies: The specified module could not be found. (Exception from HRESULT: 0x8007007E)

Click to see full output

Updating group Build in path_to\paket.dependencies
Resolving packages for group Build:
 - Fake.Core.Target 5.1.0
 - System.Data.SqlClient 4.5.1
 - Fake.Core.CommandLineParsing 5.1.0
 - Fake.Core.Context 5.1.0
 - Fake.Core.Environment 5.1.0
 - Fake.Core.FakeVar 5.1.0
 - Fake.Core.Process 5.1.0
 - Fake.Core.String 5.1.0
 - Fake.Core.Trace 5.1.0
 - System.Memory 4.5.1
 - Microsoft.Win32.Registry 4.5.0
 - System.Diagnostics.DiagnosticSource 4.5.0
 - System.Security.Principal.Windows 4.5.0
 - System.Text.Encoding.CodePages 4.5.0
 - runtime.native.System.Data.SqlClient.sni 4.5.0
 - System.Buffers 4.5.0
 - System.Reflection.TypeExtensions 4.5.0
 - FSharp.Core 4.5.0
 - System.Data.Common 4.3.0
 - System.IO.Pipes 4.3.0
 - System.Net.NameResolution 4.3.0
 - System.Net.Security 4.3.2
 - System.Security.Principal 4.3.0
 - System.Threading.Thread 4.3.0
 - System.Threading.ThreadPool 4.3.0
 - FSharp.Control.Reactive 4.1.0
 - System.Reactive.Compatibility 4.0.0
 - Microsoft.NETCore.Platforms 2.1.0
 - NETStandard.Library 2.0.3
 - Fake.IO.FileSystem 5.1.0
 - System.Security.AccessControl 4.5.0
 - System.Runtime.CompilerServices.Unsafe 4.5.1
 - runtime.win-arm64.runtime.native.System.Data.SqlClient.sni 4.4.0
 - runtime.win-x64.runtime.native.System.Data.SqlClient.sni 4.4.0
 - runtime.win-x86.runtime.native.System.Data.SqlClient.sni 4.4.0
 - System.Numerics.Vectors 4.5.0
 - System.Net.Http 4.3.3
 - runtime.native.System.Security.Cryptography.OpenSsl 4.3.2
 - System.Diagnostics.Process 4.3.0
 - System.AppContext 4.3.0
 - System.Collections.Concurrent 4.3.0
 - System.Console 4.3.1
 - System.Diagnostics.Debug 4.3.0
 - System.Diagnostics.Tools 4.3.0
 - System.Diagnostics.Tracing 4.3.0
 - System.Globalization.Calendars 4.3.0
 - System.IO.Compression 4.3.0
 - System.IO.Compression.ZipFile 4.3.0
 - System.IO.FileSystem 4.3.0
 - System.Linq 4.3.0
 - System.Linq.Expressions 4.3.0
 - System.Net.Primitives 4.3.0
 - System.ObjectModel 4.3.0
 - System.Reflection 4.3.0
 - System.Reflection.Extensions 4.3.0
 - System.Reflection.Primitives 4.3.0
 - System.Runtime.InteropServices 4.3.0
 - System.Runtime.InteropServices.RuntimeInformation 4.3.0
 - System.Runtime.Numerics 4.3.0
 - System.Security.Cryptography.Algorithms 4.3.1
 - System.Text.Encoding 4.3.0
 - System.Text.Encoding.Extensions 4.3.0
 - System.Threading 4.3.0
 - System.Threading.Timer 4.3.0
 - System.Xml.ReaderWriter 4.3.1
 - System.Xml.XDocument 4.3.0
 - System.Collections 4.3.0
 - System.Globalization 4.3.0
 - System.IO 4.3.0
 - System.Resources.ResourceManager 4.3.0
 - System.Runtime 4.3.0
 - System.Runtime.Extensions 4.3.0
 - System.Text.RegularExpressions 4.3.0
 - System.Threading.Tasks 4.3.0
 - runtime.native.System 4.3.0
 - System.IO.FileSystem.Primitives 4.3.0
 - System.Net.Sockets 4.3.0
 - System.Threading.Overlapped 4.3.0
 - runtime.native.System.Net.Security 4.3.0
 - System.Globalization.Extensions 4.3.0
 - System.Security.Cryptography.Encoding 4.3.0
 - System.Security.Cryptography.OpenSsl 4.5.0
 - System.Security.Cryptography.Primitives 4.3.0
 - System.Security.Cryptography.X509Certificates 4.3.2
 - Microsoft.Win32.Primitives 4.3.0
 - System.Runtime.Handles 4.3.0
 - System.Security.Claims 4.3.0
 - System.Reactive 4.0.0
 - System.Reactive.Core 4.0.0
 - System.Reactive.Experimental 4.0.0
 - System.Reactive.Interfaces 4.0.0
 - System.Reactive.Linq 4.0.0
 - System.Reactive.PlatformServices 4.0.0
 - System.Reactive.Providers 4.0.0
 - System.Reactive.Runtime.Remoting 4.0.0
 - System.Reactive.Windows.Forms 4.0.0
 - System.Reactive.Windows.Threading 4.0.0
 - System.Reactive.WindowsRuntime 4.0.0
 - FParsec 1.0.3
 - Microsoft.NETCore.UniversalWindowsPlatform 6.1.5
 - runtime.debian.8-x64.runtime.native.System.Security.Cryptography.OpenSsl 4.3.2
 - runtime.fedora.23-x64.runtime.native.System.Security.Cryptography.OpenSsl 4.3.2
 - runtime.fedora.24-x64.runtime.native.System.Security.Cryptography.OpenSsl 4.3.2
 - runtime.opensuse.13.2-x64.runtime.native.System.Security.Cryptography.OpenSsl 4.3.2
 - runtime.opensuse.42.1-x64.runtime.native.System.Security.Cryptography.OpenSsl 4.3.2
 - runtime.osx.10.10-x64.runtime.native.System.Security.Cryptography.OpenSsl 4.3.2
 - runtime.rhel.7-x64.runtime.native.System.Security.Cryptography.OpenSsl 4.3.2
 - runtime.ubuntu.14.04-x64.runtime.native.System.Security.Cryptography.OpenSsl 4.3.2
 - runtime.ubuntu.16.04-x64.runtime.native.System.Security.Cryptography.OpenSsl 4.3.2
 - runtime.ubuntu.16.10-x64.runtime.native.System.Security.Cryptography.OpenSsl 4.3.2
 - runtime.native.System.Security.Cryptography.Apple 4.3.1
 - System.Diagnostics.FileVersionInfo 4.3.0
 - System.IO.FileSystem.Watcher 4.3.0
 - runtime.native.System.IO.Compression 4.3.1
 - System.Reflection.Emit 4.3.0
 - System.Reflection.Emit.ILGeneration 4.3.0
 - System.Reflection.Emit.Lightweight 4.3.0
 - runtime.native.System.Net.Http 4.3.0
 - System.Runtime.InteropServices.WindowsRuntime 4.3.0
 - System.Security.Cryptography.Cng 4.5.0
 - System.Security.Cryptography.Csp 4.3.0
 - System.Threading.Tasks.Extensions 4.5.1
 - Microsoft.NETCore.Targets 2.1.0
 - runtime.osx.10.10-x64.runtime.native.System.Security.Cryptography.Apple 4.3.1
 - System.Reflection.Metadata 1.6.0
 - System.Collections.Immutable 1.5.0
Resolving packages for group Main:
Locked version resolution written to path_to\paket.lock
Starting full restore process.
run all
Building project with version: LocalBuild
Shortened DependencyGraph for Target all:
<== all

The running order is:
Group - 1
  - all
Starting target 'all'
NoDescription
Finished (Failed) 'all' in 00:00:00.0685860

---------------------------------------------------------------------
Build Time Report
---------------------------------------------------------------------
Target     Duration
------     --------
all        00:00:00.0610577   (The type initializer for 'System.Data.SqlClient.TdsParser' threw an exception.)
Total:     00:00:00.1896509
Status:    Failure
---------------------------------------------------------------------
Script reported an error:
-> BuildFailedException: Target 'all' failed.
-> One or more errors occurred. (The type initializer for 'System.Data.SqlClient.TdsParser' threw an exception.)
-> TypeInitializationException: The type initializer for 'System.Data.SqlClient.TdsParser' threw an exception.
-> TypeInitializationException: The type initializer for 'System.Data.SqlClient.SNILoadHandle' threw an exception.
-> DllNotFoundException: Unable to load DLL 'sni.dll' or one of its dependencies: The specified module could not be found. (Exception from HRESULT: 0x8007007E)
Hint: To further diagnose the problem you can run fake in verbose mode `fake -v run ...` or set the 'FAKE_DETAILED_ERRORS' environment variable to 'true'
Performance:
 - Cli parsing: 238 milliseconds
 - Packages: 27 seconds
   - Resolver: 24 seconds (2 runs)
      - Runtime: 3 seconds
      - Blocked (retrieving package details): 17 seconds (34 times)
      - Blocked (retrieving package versions): 3 seconds (5 times)
      - Not Blocked (retrieving package details): 92 times
      - Not Blocked (retrieving package versions): 121 times
   - Disk IO: 33 milliseconds
   - Average Request Time: 94 milliseconds
   - Number of Requests: 255
   - Creating Runtime Graph: 118 milliseconds
   - Retrieve Assembly List: 643 milliseconds
 - Script analyzing: 21 milliseconds
 - Script running: 462 milliseconds
 - Script cleanup: 0 milliseconds
 - Runtime: 28 seconds

Known workarounds

Providing --additional-deps to dotnet cli:

dotnet --additional-deps extra.deps.json .fake\.store\fake-cli\5.0.0\fake-cli\5.0.0\tools\netcoreapp2.1\any\fake-cli.dll build

I've built an otherwise empty F# console application with System.Data.SqlClient reference and removed everything but sni related packages/libraries from its deps.json file.

Click to see full extra.deps.json file

{
  "runtimeTarget": {
    "name": ".NETCoreApp,Version=v2.1"
  },
  "targets": {
    ".NETCoreApp,Version=v2.1": {
      "runtime.native.System.Data.SqlClient.sni/4.4.0": {
        "dependencies": {
          "runtime.win-arm64.runtime.native.System.Data.SqlClient.sni": "4.4.0",
          "runtime.win-x64.runtime.native.System.Data.SqlClient.sni": "4.4.0",
          "runtime.win-x86.runtime.native.System.Data.SqlClient.sni": "4.4.0"
        }
      },
      "runtime.win-arm64.runtime.native.System.Data.SqlClient.sni/4.4.0": {
        "runtimeTargets": {
          "runtimes/win-arm64/native/sni.dll": {
            "rid": "win-arm64",
            "assetType": "native",
            "fileVersion": "4.6.25512.1"
          }
        }
      },
      "runtime.win-x64.runtime.native.System.Data.SqlClient.sni/4.4.0": {
        "runtimeTargets": {
          "runtimes/win-x64/native/sni.dll": {
            "rid": "win-x64",
            "assetType": "native",
            "fileVersion": "4.6.25512.1"
          }
        }
      },
      "runtime.win-x86.runtime.native.System.Data.SqlClient.sni/4.4.0": {
        "runtimeTargets": {
          "runtimes/win-x86/native/sni.dll": {
            "rid": "win-x86",
            "assetType": "native",
            "fileVersion": "4.6.25512.1"
          }
        }
      }
    }
  },
  "libraries": {
    "runtime.native.System.Data.SqlClient.sni/4.4.0": {
      "type": "package",
      "serviceable": true,
      "sha512": "sha512-A8v6PGmk+UGbfWo5Ixup0lPM4swuSwOiayJExZwKIOjTlFFQIsu3QnDXECosBEyrWSPryxBVrdqtJyhK3BaupQ==",
      "path": "runtime.native.system.data.sqlclient.sni/4.4.0",
      "hashPath": "runtime.native.system.data.sqlclient.sni.4.4.0.nupkg.sha512"
    },
    "runtime.win-arm64.runtime.native.System.Data.SqlClient.sni/4.4.0": {
      "type": "package",
      "serviceable": true,
      "sha512": "sha512-LbrynESTp3bm5O/+jGL8v0Qg5SJlTV08lpIpFesXjF6uGNMWqFnUQbYBJwZTeua6E/Y7FIM1C54Ey1btLWupdg==",
      "path": "runtime.win-arm64.runtime.native.system.data.sqlclient.sni/4.4.0",
      "hashPath": "runtime.win-arm64.runtime.native.system.data.sqlclient.sni.4.4.0.nupkg.sha512"
    },
    "runtime.win-x64.runtime.native.System.Data.SqlClient.sni/4.4.0": {
      "type": "package",
      "serviceable": true,
      "sha512": "sha512-38ugOfkYJqJoX9g6EYRlZB5U2ZJH51UP8ptxZgdpS07FgOEToV+lS11ouNK2PM12Pr6X/PpT5jK82G3DwH/SxQ==",
      "path": "runtime.win-x64.runtime.native.system.data.sqlclient.sni/4.4.0",
      "hashPath": "runtime.win-x64.runtime.native.system.data.sqlclient.sni.4.4.0.nupkg.sha512"
    },
    "runtime.win-x86.runtime.native.System.Data.SqlClient.sni/4.4.0": {
      "type": "package",
      "serviceable": true,
      "sha512": "sha512-YhEdSQUsTx+C8m8Bw7ar5/VesXvCFMItyZF7G1AUY+OM0VPZUOeAVpJ4Wl6fydBGUYZxojTDR3I6Bj/+BPkJNA==",
      "path": "runtime.win-x86.runtime.native.system.data.sqlclient.sni/4.4.0",
      "hashPath": "runtime.win-x86.runtime.native.system.data.sqlclient.sni.4.4.0.nupkg.sha512"
    }
  }
}

Related information

• Windows 10 x64
• FAKE 5 as installed by dotnet new fake
• .NET Core 2.1

[matthid]

There are two problems here.

1. How to find the dlls. I think this is already implemented in paket (runtime dependency resolution, disabled by default). So I think this is just a matter of enabling and potentially fixing bugs. The only problem is that this is slow so we might make this opt-in...

2. How to use the information at runtime. I don’t know if and how this is possible or what apis need to be used...

So we need to figure out 2.
In order to do that you need to find a workaround without “—additional-deps”...

[BlythMeister]

we are going to need to come up with a solution for this sooner rather than later.

The new Fake.Sql.SqlServer module doesn't work because of this exact issue :(

[BlythMeister]

also, the workaround only works when using fake-cli.
when using dotnet-fake you can't pass additional-deps :(

So i've hit a brick wall trying to solve this :(

[matthid]

@BlythMeister The question is if we can achieve the same not at start of the dotnet core runtime but later when the code is already running. Another way would be to "restart" fake itself with the proper arguments once this situation is detected. This probably means we need to generate the additional-deps file ...

[matthid]

Ok I did a bit of digging and apparently restarting ourself is the only option right now:

The only problem that still exists in net core 2.0 is if you are trying to dynamically load a netcoreapp assembly that brings in other or different runtime dependencies.

https://github.com/dotnet/corefx/issues/21982#issuecomment-382338058

There seems to be an ongoing effort to support this scenario (If I actually understand that correctly): dotnet/designs#47

[matthid]

Ok I didn't found any workaround from within the script code, but apparently there is a method we can override to customize native dll resolution: https://github.com/dotnet/coreclr/issues/12103#issuecomment-397739206
I'll play around with this a bit later (but please anyone feel free to do some experiments as well)

[matthid]

Just in case someone wants to step in. I tried the following (on windows x64):

• Call Kernel32.LoadLibrary before running the target
• Call Kernel32.AddDllPath (or AddDllBasePath) before running the target
• Set current directory to the directory containing sni.dll
• Set the PATH variable to a directory containing sni.dll

None worked so far (I'm not sure why they wouldn't)

[panesofglass]

Is the only current work-around for issuing SQL commands to shell out to sqlcmd.exe?

[csmager]

I did a little digging into this. I'm doing my digging on macOS in the context of #2320 - I'm using the build script I included in that issue. I modified the custom AssemblyLoadContext:

https://github.com/fsharp/FAKE/blob/cecc72166334a3804c18a4c0ccd4911c94e58e45/src/app/Fake.Runtime/CoreCache.fs#L331-L335

To this, hard coding the path for the relevant native dll for my platform:

type FakeLoadContext (printDetails:Trace.VerboseLevel, dependencies:AssemblyInfo list) =
  inherit AssemblyLoadContext()
  let allReferences = dependencies
  override x.Load(assem:AssemblyName) =
       findAndLoadInRuntimeDepsCached x assem printDetails allReferences
  override x.LoadUnmanagedDll(unmanagedAssemblyName) =
      if unmanagedAssemblyName = "SQLite.Interop.dll" then
        base.LoadUnmanagedDllFromPath("/Users/cmager/.nuget/packages/system.data.sqlite.core/1.0.110/runtimes/osx-x64/native/netstandard2.0/SQLite.Interop.dll")
      else
        base.LoadUnmanagedDll(unmanagedAssemblyName)

And now my script runs fine. @matthid does this help with 'step 2' you referred to above? If we had some map of native runtime deps to paths (based on platform), then this looks solvable.

[matthid]

Yes with this information it should be quite straightforward to solve as the list of native libraries is provided by paket. sadly I can't commit to a specific time frame for a fix for this...

[matthid]

Now we only need to wait for fsprojects/Paket#3593 to be released for this to work

[matthid]

I'm closing this: It will be part of the 5.14 release. Note: You need to update the runner to get this feature. Updating your build will not be enough.

[csmager]

Thanks for fixing, looking forward to getting my FAKE 4 scripts migrated now it works! It seems fsprojects/Paket#3593 has been merged & released now, so hopefully 5.14 isn't too far away.

[matthid]

@csmager Yes, 5.14 release is triggered.
There is one problem, however:

  Error: Script reported an error:
  -> BuildFailedException: Target 'Default' failed.
     StackTrace:
          at Fake.Core.TargetModule.raiseIfError(OptionalTargetContext context) in D:\a\1\s\src\app\Fake.Core.Target\Target.fs:line 820
          at Fake.Core.TargetModule.runOrDefault(String defaultTarget) in D:\a\1\s\src\app\Fake.Core.Target\Target.fs:line 968
          at <StartupCode$build_D4BA2A145FCE452104A4064500301D324DD51611A77E88C264B76CFD9928D13C>.$Build$fsx.main@() in /home/travis/build/fsharp/FAKE/integrationtests/i002007-native-libs/temp/build.fsx:line 23
  -> One or more errors occurred. (Unable to find an entry point named 'sqlite3_config' in shared library 'SQLite.Interop.dll'.)
  -> EntryPointNotFoundException: Unable to find an entry point named 'sqlite3_config' in shared library 'SQLite.Interop.dll'.
     StackTrace:
          at System.Data.SQLite.UnsafeNativeMethods.sqlite3_config_log(SQLiteConfigOpsEnum op, SQLiteLogCallback func, IntPtr pvUser)
          at System.Data.SQLite.SQLite3.SetLogCallback(SQLiteLogCallback func)
          at System.Data.SQLite.SQLiteLog.Initialize(String className)
          at System.Data.SQLite.SQLiteConnection..ctor(String connectionString, Boolean parseViaFramework)
          at Build.openConn(String path) in /home/travis/build/fsharp/FAKE/integrationtests/i002007-native-libs/temp/build.fsx:line 13
          at Build.clo@16.Invoke(TargetParameter _arg1) in /home/travis/build/fsharp/FAKE/integrationtests/i002007-native-libs/temp/build.fsx:line 17
          at Fake.Core.TargetModule.runSimpleInternal(TargetContext context, Target target) in D:\a\1\s\src\app\Fake.Core.Target\Target.fs:line 201. Actual value was false but had expected it to be true.

This only happens on linux/unix therefore I don't think this is a bug in FAKE. I think if you need unix/linux support you need to report a bug to them. If you are already on it you can tell them their package is packaged incorrectly as well :)

[csmager]

Thanks. It works fine for me on Windows, but I can repro that error on macOS. My hack to get it to work was on macOS & it did seem to work. I'll dig into it and see if I can work out which of SQLite, Paket or FAKE is doing something wrong!

Re the packaging, is there do you happen to know where it's how this is supposed to be packaged is documented?

[csmager]

Ok, the logging shows the file path is correct:

Trying to resolve native library: SQLite.Interop.dll
Redirect native library 'SQLite.Interop.dll' to known path: /Users/cmager/.nuget/packages/system.data.sqlite.core/1.0.111/runtimes/osx-x64/native/netstandard2.0/SQLite.Interop.dll
Redirect native library 'SQLite.Interop.dll' to previous loaded library '/Users/cmager/.nuget/packages/system.data.sqlite.core/1.0.111/runtimes/osx-x64/native/netstandard2.0/SQLite.Interop.dll'

The only difference I guessed was the caching. So I removed it by changing the FakeLoadContext code to this:

override x.LoadUnmanagedDll(unmanagedDllName) =
    findAndLoadUnmanagedInRuntimeDeps x.LoadUnmanagedDllFromPathHelper unmanagedDllName printDetails nativeLibraries |> fst

And now it works fine. So it seems caching the IntPtr value breaks it.

[matthid]

Oo that was unexpected, so we should just remove the dictionary and load it every time (we could cache the full path instead)?

[matthid]

My hack to get it to work was on macOS & it did seem to work.

Thanks. I was on the completely wrong path here.

So it seems caching the IntPtr value breaks it.

Indeed, thanks for the pointers (pun intended)... I changed the logic slightly and it looks green. I'll wait on the build and start the 5.14.1 release :)

[csmager]

Thanks. Can't say I fully understand why it didn't work, there seems to be a lack of docs around AssemblyLoadContext. I can only assume that the framework handles the lifetime of that handle and it's not valid at the time of the second call.