netty-tcnative-boringssl-static fails to load after version 2.0.48

[spencejackson]

Starting in 2.0.49, I am seeing:

OpenSsl.isAvailable() returns false

OpenSsl.unavailabilityCause() returns
[io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:114), io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:705), io.netty.handler.ssl.OpenSsl.(OpenSsl.java:146), io.baffle.shield.sql.mysql.BaffleShield.run(BaffleShield.java:349)]

[temawi]

I have also been impacted by this while trying to upgrade grpc-java to the latest netty. It looks like since netty-tcnative-boringssl-static 2.0.49 the jar no longer contains the native libraries and at least Gradle and Bazel are having a hard time including the dependent libraries that now have them.

In our Gradle build I was able to work around this by including all the individual libraries (e.g. "io.netty:netty-tcnative-boringssl-static:2.0.51.Final:linux-x86_64") excplicitly.

Bazel does not seem to like this approach though and reports a cyclical dependency:

ERROR: /tmpfs/tmp/bazel/external/maven/BUILD:548:11: in jvm_import rule @maven//:io_netty_netty_tcnative_boringssl_static_osx_aarch_64: cycle in dependency graph:
//netty/shaded:shaded
@io_netty_netty_tcnative_boringssl_static_linux_x86_64//jar:jar
@maven//:io_netty_netty_tcnative_boringssl_static_linux_x86_64
.-> @maven//:io_netty_netty_tcnative_boringssl_static_osx_aarch_64
| @maven//:io_netty_netty_tcnative_boringssl_static_linux_aarch_64
`-- @maven//:io_netty_netty_tcnative_boringssl_static_osx_aarch_64

@ejona86

[ejona86]

@spencejackson, are you using Gradle? It seems things may work okay for Maven, but not Gradle.

Classifiers are a murky part of Maven's package format. It isn't surprising to me that different systems don't all work identically here. It may be best to restore the uber jar for the native/ files, but leave the class files split out.

[spencejackson]

Yes, I'm using Gradle.

[fedorka]

We're also using Gradle and are impacted by the change. We were packaging the uber-jar as part of our platform IPC libraries so we can support both our deployed servers and the wide variety of developer environments.

Edit: We are working around this by explicitly bringing in each of the classifiers, it's working well in our initial tests 😄

[sannam888]

@ejona86 I doubt that the problem is related to Gradle. From what I can see, the netty-tcnative-boringssl-static uber jar which used to contain the native libraries for all the platforms is now empty! Hence anyone depending on the uber jar will now fail to load the native library.

The most likely commit that broke this is: 6fb5b01

Note that libraries that depend on netty-tcnative-boringssl-static (eg. Spring) have dependencies on the uber jar because they don't know what platform their users are going to be on.

[spencejackson]

I agree. After 2.0.48 the size of the jar takes a nosedive.

@ejona86 I doubt that the problem is related to Gradle. From what I can see, the netty-tcnative-boringssl-static uber jar which used to contain the native libraries for all the platforms is now empty! Hence anyone depending on the uber jar will now fail to load the native library.

The most likely commit that broke this is: 6fb5b01

Note that libraries that depend on netty-tcnative-boringssl-static (eg. Spring) have dependencies on the uber jar because they don't know what platform their users are going to be on.

[ejona86]

From what I can see, the netty-tcnative-boringssl-static uber jar which used to contain the native libraries for all the platforms is now empty!

But the pom now depends on the various platform-specific classifiers. So the other binaries are getting brought it from the dependency system. It seems it was likely tested with Maven and got working, but Gradle behaves differently.

[Zetten]

I think a related issue is being thrown up by 6da7a21 (included in io.netty:netty-tcnative-boringssl-static:2.0.51.Final) which changes the Automatic-Module-Name in the manifest of the former uberjar, and hardcodes it to osx-x86_64:

$ curl -sL https://repo.maven.apache.org/maven2/io/netty/netty-tcnative-boringssl-static/2.0.51.Final/netty-tcnative-boringssl-static-2.0.51.Final.jar | zcat -q - | grep Automatic-Module-Name
Automatic-Module-Name: io.netty.tcnative.openssl.osx-x86_64

This is causing problems in my bazel build where javac is failing to handle this transitive dep on the classpath:

error: cannot determine module name for bazel-out/k8-fastbuild/bin/external/maven/v1/https/repo.maven.apache.org/maven2/io/netty/netty-tcnative-boringssl-static/2.0.51.Final/header_netty-tcnative-boringssl-static-2.0.51.Final.jar

[ejona86]

@Zetten, yes, we saw that issue as well, but there was already some discussion about that. @temawi, might have a link. When we semi-worked around this issue Java 11 execution failed but Java 8 was okay. But it seems they are two separate issues.

[liuchong]

Same issue here in clojure app build with leiningen:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.base/java.lang.reflect.Method.copy(Method.java:161)
	at java.base/java.lang.reflect.ReflectAccess.copyMethod(ReflectAccess.java:102)
	at java.base/jdk.internal.reflect.ReflectionFactory.copyMethod(ReflectionFactory.java:309)
	at java.base/java.lang.Class.copyMethods(Class.java:3627)
	at java.base/java.lang.Class.getMethods(Class.java:2019)
	at clojure.lang.Reflector.getMethods(Reflector.java:498)
	at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:436)
	at leiningen.core.pedantic$all_paths$iter__6753__6759$fn__6760.invoke(pedantic.clj:110)
	at clojure.lang.LazySeq.sval(LazySeq.java:42)
	at clojure.lang.LazySeq.seq(LazySeq.java:58)
	at clojure.lang.ChunkedCons.chunkedNext(ChunkedCons.java:59)
	at clojure.lang.ChunkedCons.next(ChunkedCons.java:43)
	at clojure.lang.RT.next(RT.java:709)
	at clojure.core$next__5371.invokeStatic(core.clj:64)
	at clojure.core$dorun.invokeStatic(core.clj:3142)
	at clojure.core$doall.invokeStatic(core.clj:3148)
	at clojure.core$doall.invoke(core.clj:3148)
	at leiningen.core.pedantic$all_paths.invokeStatic(pedantic.clj:113)
	at leiningen.core.pedantic$all_paths.invoke(pedantic.clj:100)
	at leiningen.core.pedantic$transform_graph.invokeStatic(pedantic.clj:123)
	at leiningen.core.pedantic$transform_graph.invoke(pedantic.clj:115)
	at leiningen.core.pedantic$use_transformer$reify__6787.transformGraph(pedantic.clj:161)
	at org.eclipse.aether.internal.impl.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:275)
	at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:307)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167)
	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:102)
	at cemerick.pomegranate.aether$resolve_dependencies_STAR_.invokeStatic(aether.clj:806)
	at cemerick.pomegranate.aether$resolve_dependencies_STAR_.doInvoke(aether.clj:707)

[ejona86]

@liuchong, did you add that comment to the wrong github issue? That error seems to have no relation with this issue.

[liuchong]

Not sure, but my fail was also start from after 2.0.48, I thinks should be related issue although the error message seems not.

[temawi]

FYI - this issue is now mentioned on the netty-tcnative site. There is a "Gradle and Bazel" section that states:

The maven packaging changed a bit in version 2.0.49.Final. The changes were fine for Maven users, but causes some trouble for Gradle and Bazel users. For Gradle, the solution is to explicitly declare the tcnative dependencies with classifiers. For Bazel, the solution is to use a Bazel version that contain this fix: bazel-contrib/rules_jvm_external#687

I have a working Gradle solution for grpc-java in grpc/grpc-java#9027, but I still need to wait for a new Bazel build with the mentioned fix.

[KAITYOZHE]

why netty-tcnative-boringssl-static-1.0.48 to netty-tcnative-boringssl-static-1.0.49 is problem ?
go up minor version Shouldn't A problem occurs.

java.lang.IllegalStateException: Could not find TLS ALPN provider; no working netty-tcnative, Conscrypt, or Jetty NPN/ALPN available
at io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:246) ~[grpc-netty-1.39.0.jar:1.39.0]
at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:146) ~[grpc-netty-1.39.0.jar:1.39.0]
at io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:95) ~[grpc-netty-1.39.0.jar:1.39.0]

[snalli]

Why does netty do this (all the time !) ?!!!

[normanmaurer]

I think this should be fixed