Linux: support broader range of distros/architectures by trying to load packed native libraries. #1714

tmds · 2019-09-09T09:48:13Z

LibGit2Sharp includes a number of native libraries that are built on different OSes.
These native libraries have a dependency on OpenSSL and the system c-library.

On x64, in case the Linux flavor is not known, a fallback is performed to the 'linux-x64' native library.
This library is built with a dependency on OpenSSL 1.0.
OpenSSL 1.0 is deprecated by OpenSSL 1.1, so on more recent version of Linux, the 'linux-x64' library fails to load.

On arm64, native libraries are currently included for debian.* (OpenSSL 1.1) and ubuntu.18.04 (OpenSSL 1.0).
Loading on other distros will fail.

In both cases LibGit2Sharp is probably including a library that works, but the default resolution logic
is not able to find it.

By using the 'NativeLibrary' class (.NET Core 3.0) we can extend the resolution logic, and try to load
the other native libraries that are packed with LibGit2Sharp.

…ad packed native libraries. LibGit2Sharp includes a number of native libraries that are built on different OSes. These native libraries have a dependency on OpenSSL and the system c-library. On x64, in case the Linux flavor is not known, a fallback is performed to the 'linux-x64' native library. This library is built with a dependency on OpenSSL 1.0. OpenSSL 1.0 is deprecated by OpenSSL 1.1, so on more recent version of Linux, the 'linux-x64' library fails to load. On arm64, native libraries are currently included for debian.* (OpenSSL 1.1) and ubuntu.18.04 (OpenSSL 1.0). Loading on other distros will fail. In both cases LibGit2Sharp is probably including a library that works, but the default resolution logic is not able to find it. By using the 'NativeLibrary' class (.NET Core 3.0) we can extend the resolution logic, and try to load the other native libraries that are packed with LibGit2Sharp.

tmds · 2019-09-09T09:49:51Z

cc @omajid @dagood @crummel

ethomson · 2019-09-09T10:02:36Z

Jesus, LibGit2Sharp needs to 🔥 the dependency on OpenSSL.

tmds · 2019-09-09T10:48:00Z

LibGit2Sharp needs to 🔥 the dependency on OpenSSL.

The dependency comes from libgit2.

ethomson · 2019-09-09T10:53:13Z

LibGit2Sharp needs to 🔥 the dependency on OpenSSL.

The dependency comes from libgit2.

Yes, I'm aware. But libgit2 can use myriad options for HTTPS, including a custom one.

tmds · 2019-09-09T12:00:22Z

But libgit2 can use myriad options for HTTPS, including a custom one.

That's a nice feature, and it will solve the main reason for load failures.

tmds · 2019-09-09T15:04:38Z

CI on Windows has a failed test:

D:\a\1\s>dotnet test "D:\a\1\s\LibGit2Sharp.Tests/LibGit2Sharp.Tests.csproj" -f netcoreapp2.1 --no-restore --no-build 
Test run for D:\a\1\s\bin\LibGit2Sharp.Tests\Release\netcoreapp2.1\LibGit2Sharp.Tests.dll(.NETCoreApp,Version=v2.1)
Microsoft (R) Test Execution Command Line Tool Version 15.9.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
[xUnit.net 00:00:11.42]     LibGit2Sharp.Tests.CloneFixture.CanInspectCertificateOnClone(url: "git@github.com:libgit2/TestGitRepository.git", hostname: "github.com", certType: typeof(LibGit2Sharp.CertificateSsh)) [SKIP]
Skipped  LibGit2Sharp.Tests.CloneFixture.CanInspectCertificateOnClone(url: "git@github.com:libgit2/TestGitRepository.git", hostname: "github.com", certType: typeof(LibGit2Sharp.CertificateSsh))
[xUnit.net 00:00:15.26]     LibGit2Sharp.Tests.CloneFixture.CanCloneWithCredentials [SKIP]
Skipped  LibGit2Sharp.Tests.CloneFixture.CanCloneWithCredentials
[xUnit.net 00:00:22.40]     LibGit2Sharp.Tests.RepositoryFixture.CanListRemoteReferencesWithCredentials [SKIP]
Skipped  LibGit2Sharp.Tests.RepositoryFixture.CanListRemoteReferencesWithCredentials
[xUnit.net 00:00:32.80]     LibGit2Sharp.Tests.FetchFixture.CanFetchIntoAnEmptyRepositoryWithCredentials [SKIP]
Skipped  LibGit2Sharp.Tests.FetchFixture.CanFetchIntoAnEmptyRepositoryWithCredentials
[xUnit.net 00:01:07.51]     LibGit2Sharp.Tests.StageFixture.CanStageANewFileWithAFullPath(ignorecase: True) [SKIP]
Skipped  LibGit2Sharp.Tests.StageFixture.CanStageANewFileWithAFullPath(ignorecase: True)
[xUnit.net 00:01:16.44]     LibGit2Sharp.Tests.DiffTreeToTreeFixture.RenameDetectionObeysConfigurationSetting [SKIP]
Skipped  LibGit2Sharp.Tests.DiffTreeToTreeFixture.RenameDetectionObeysConfigurationSetting
[xUnit.net 00:02:51.08]     LibGit2Sharp.Tests.ResetHeadFixture.SoftResetSetsTheHeadToTheSpecifiedCommit [FAIL]
Failed   LibGit2Sharp.Tests.ResetHeadFixture.SoftResetSetsTheHeadToTheSpecifiedCommit
Error Message:
 Assert.InRange() Failure
Range:  (9/9/2019 3:00:24 PM +00:00 - 9/9/2019 3:00:24 PM +00:00)
Actual: 9/9/2019 3:00:23 PM +00:00
Stack Trace:
   at LibGit2Sharp.Tests.TestHelpers.BaseFixture.AssertRefLogEntry(IRepository repo, String canonicalName, String message, ObjectId from, ObjectId to, Identity committer, DateTimeOffset before) in D:\a\1\s\LibGit2Sharp.Tests\TestHelpers\BaseFixture.cs:line 471
   at LibGit2Sharp.Tests.ResetHeadFixture.AssertSoftReset(Func`2 branchIdentifierRetriever, Boolean shouldHeadBeDetached, Func`2 expectedHeadNameRetriever) in D:\a\1\s\LibGit2Sharp.Tests\ResetHeadFixture.cs:line 128
[xUnit.net 00:02:57.00]     LibGit2Sharp.Tests.GlobalSettingsFixture.LoadFromSpecifiedPath(architecture: "x86") [SKIP]
[xUnit.net 00:02:57.01]     LibGit2Sharp.Tests.GlobalSettingsFixture.LoadFromSpecifiedPath(architecture: "x64") [SKIP]
Skipped  LibGit2Sharp.Tests.GlobalSettingsFixture.LoadFromSpecifiedPath(architecture: "x86")
Skipped  LibGit2Sharp.Tests.GlobalSettingsFixture.LoadFromSpecifiedPath(architecture: "x64")
[xUnit.net 00:03:11.74]     LibGit2Sharp.Tests.NetworkFixture.CanListRemoteReferencesWithCredentials [SKIP]
Skipped  LibGit2Sharp.Tests.NetworkFixture.CanListRemoteReferencesWithCredentials

tmds · 2019-09-10T17:29:08Z

Independent of getting rid of the OpenSSL dependency, this change has value. For example, when new releases of Alpine are made (which the rid graph doesn't yet map to linux-musl-x64), they fall back to linux-x64 which doesn't work.

ethomson · 2019-09-10T18:05:35Z

Sure, sorry, I was idly speculating / complaining, I wasn't suggesting that any other changes should preclude merging it. But @bording understands this stuff much better than I, so I'll leave it to him.

tmds · 2019-09-16T08:00:39Z

@bording can you take a look?

We've hit this a few times when bringing .NET Core to new versions of Fedora (which causes the mapping to fedora-x64 to fail).
And now also when trying out .NET Core on arm64 on Fedora (because there isn't a fedora-arm64).

bording · 2019-09-16T14:57:05Z

@tmds I've been rather busy recently and haven't had much spare time to devote to LibGit2Sharp recently, but I'll try and take a look at this soon.

I'm curious where you've hit problems with this, though. Where in .NET Core are you running into a LibGit2Sharp dependency? It was my understanding that that the only place was in the Source Link packages, but those have since removed LibGit2Sharp entirely.

tmds · 2019-09-16T15:22:26Z

dotnet/source-build#1083 describes the issues with source-build.

The issue applies to any app using libgit2sharp that is meant to work on a broad range of linux-x64/linux-arm64 distros.

bording · 2019-09-21T19:36:46Z

So this is an interesting change, and I had been following the NativeLibrary stuff in .NET Core 3.0 with an idea towards being able to leverage it in some way.

The NativeBinaries package has always had to walk the tightrope of ensuring as many distros as possible are covered, without also completely exploding the size of the package by building a library for each possible RID. That means that I've had to pay a lot of attention to the RID graph to figure out how a given distro gets mapped in there. And, as you've observed, it's a moving target, so we're always having to play catch up.

I think what this really demonstrates is that .NET Core's RID mechanism needs improvement, but I know there are already open issues about that.

As a first step, I think it makes sense to make this change. Eventually, I'd be inclined to add a netcoreapp3.0 target instead of using reflection, but that can always be done later.

bording · 2019-09-21T20:40:03Z

This has been published on NuGet.org as 0.27.0-preview-0020.

tmds · 2019-09-24T12:42:03Z

This has been published on NuGet.org as 0.27.0-preview-0020.

tmds · 2019-09-24T12:47:42Z

This has been published on NuGet.org as 0.27.0-preview-0020.

I tinkered with my /etc/os-release and unknown future Fedora versions can use LibGit2Sharp!

I think what this really demonstrates is that .NET Core's RID mechanism needs improvement, but I know there are already open issues about that.

I agree. This is the main one: NuGet/Home#5862.

LibGit2Sharp 0.27.0-preview-0020 improves support for a number of architectures and linux distributions: libgit2/libgit2sharp#1714 This makes it possible to use arcade (via source-build) on RHEL on arm64.

tanmay-18 · 2020-04-21T09:21:43Z

This has been published on NuGet.org as 0.27.0-preview-0020.

Issue still persists with 'git2-7ce88e6' dll not found.

tmds mentioned this pull request Sep 9, 2019

Not possible to build source-build release/3.0 on new distributions dotnet/source-build#1083

Closed

Fix netfx build

cdab62d

bording approved these changes Sep 21, 2019

View reviewed changes

bording merged commit 6740402 into libgit2:master Sep 21, 2019

This was referenced Oct 5, 2019

Add linux-arm64 RID builds libgit2/libgit2sharp.nativebinaries#99

Closed

System.DllNotFoundException: Unable to load shared library 'git2-8e0b172' dotnet/source-build#1125

Closed

omajid mentioned this pull request Oct 9, 2019

Bump LibGit2Sharp version dotnet/arcade-services#694

Merged

qmfrederik mentioned this pull request Oct 28, 2019

Merge [v3.0] -> [master] dotnet/Nerdbank.GitVersioning#400

Merged

tmds mentioned this pull request Oct 28, 2019

Fix native library loading on Linux #1732

Closed

This was referenced Nov 11, 2019

Problem in picking up libgit2-7ce88e6 in fedora29 #1730

Open

Install darc rather than building it via arcade-services dotnet/source-build#1315

Merged

crummel mentioned this pull request Nov 19, 2019

Remove obsolete arcade-services submodule and upgrade to netcoreapp3.0 version of Darc. dotnet/source-build#1368

Merged

qmfrederik mentioned this pull request Jan 23, 2020

Visual Studio Code/OmniSharp on Ubuntu 18.04: DllNotFoundException dotnet/Nerdbank.GitVersioning#417

Closed

This was referenced Mar 30, 2020

Support Ubuntu 20.04 (Focal) #1747

Closed

Unable to load shared library 'git2-7ce88e6' or one of its dependencies. #1703

Closed

ashishdhingra mentioned this pull request Aug 10, 2020

Unable to load shared library 'git2-ef5a385' or one of its dependencies aws/aws-lambda-dotnet#716

Closed

qmfrederik mentioned this pull request Sep 8, 2020

Consider using a managed Git backend dotnet/Nerdbank.GitVersioning#505

Closed

omajid mentioned this pull request Nov 9, 2020

build.sh causes segfault on Alpine Linux just after running Darc dotnet/source-build#1868

Closed

omajid mentioned this pull request Apr 6, 2022

Simplify RID Model dotnet/designs#260

Open

premun mentioned this pull request Jun 30, 2023

Bump LibGit2Sharp to 0.27.2 dotnet/arcade-services#2728

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Linux: support broader range of distros/architectures by trying to load packed native libraries. #1714

Linux: support broader range of distros/architectures by trying to load packed native libraries. #1714

tmds commented Sep 9, 2019

tmds commented Sep 9, 2019

ethomson commented Sep 9, 2019

tmds commented Sep 9, 2019

ethomson commented Sep 9, 2019

tmds commented Sep 9, 2019

tmds commented Sep 9, 2019

tmds commented Sep 10, 2019

ethomson commented Sep 10, 2019

tmds commented Sep 16, 2019

bording commented Sep 16, 2019

tmds commented Sep 16, 2019 •

edited

Loading

bording commented Sep 21, 2019

bording commented Sep 21, 2019

tmds commented Sep 24, 2019

tmds commented Sep 24, 2019 •

edited

Loading

tanmay-18 commented Apr 21, 2020

Linux: support broader range of distros/architectures by trying to load packed native libraries. #1714

Linux: support broader range of distros/architectures by trying to load packed native libraries. #1714

Conversation

tmds commented Sep 9, 2019

tmds commented Sep 9, 2019

ethomson commented Sep 9, 2019

tmds commented Sep 9, 2019

ethomson commented Sep 9, 2019

tmds commented Sep 9, 2019

tmds commented Sep 9, 2019

tmds commented Sep 10, 2019

ethomson commented Sep 10, 2019

tmds commented Sep 16, 2019

bording commented Sep 16, 2019

tmds commented Sep 16, 2019 • edited Loading

bording commented Sep 21, 2019

bording commented Sep 21, 2019

tmds commented Sep 24, 2019

tmds commented Sep 24, 2019 • edited Loading

tanmay-18 commented Apr 21, 2020

tmds commented Sep 16, 2019 •

edited

Loading

tmds commented Sep 24, 2019 •

edited

Loading