io.quarkus.it.kafka.streams.KafkaStreamsTest.testKafkaStreams fail to locate JNI library on Z #6667

smlambert · 2020-01-21T01:43:26Z

Describe the bug
When trying to run quarkus tests on zlinux architecture at the AdoptOpenJDK project, the io.quarkus.it.kafka.streams.KafkaStreamsTest.testKafkaStreams test fails to launch Quarkus due to a missing JNI library

TestInstanceFactory [io.quarkus.test.junit.QuarkusTestExtension] failed to instantiate test class [io.quarkus.it.kafka.streams.KafkaStreamsTest]: Failed to start quarkus ... Caused by: java.lang.RuntimeException: librocksdbjni-linuxs390x.so was not found inside JAR.

Expected behavior
Expect to be able to run all quarkus tests on zlinux

Actual behavior
This was noticed when trying to run quarkus tests in JVM mode, using a nightly jdk8 openj9 docker image from AdoptOpenJDK. Test results can be seen here: https://ci.adoptopenjdk.net/view/work%20in%20progress/job/Grinder_Sandbox/1079/testReport/
From console output of that test run:
21:28:32 [�[1;34mINFO�[m] Quarkus - Integration Tests - Kafka Streams ........ �[1;31mFAILURE�[m [ 7.076 s]

To Reproduce
Steps to reproduce the behavior:

Run on s390x machine, based off openj9 Dockerfile
Follow instructions to set up and run quarkus tests (from contribution guide)
As you will hit the mongodb issues reported in Issue 3423, you will want to exclude the mongodb tests, or run "Quarkus - Integration Tests - Kafka Streams" tests (and dependencies to see the issue

Environment (please complete the following information):
21:13:28 Running quarkus_java_test ...
21:13:28 "/home/jenkins/workspace/Grinder_Sandbox/openjdkbinary/j2sdk-image/bin/java" -version
21:13:28 openjdk version "1.8.0_242"
21:13:28 OpenJDK Runtime Environment (build 1.8.0_242-202001160515-b07)
21:13:28 Eclipse OpenJ9 VM (build master-99a64d22d, JRE 1.8.0 Linux s390x-64-Bit Compressed References 20200116_498 (JIT enabled, AOT enabled)
21:13:28 OpenJ9 - 99a64d22d
21:13:28 OMR - 947946e7d
21:13:28 JCL - e6095d9adfe based on jdk8u242-b07)

Based on the failure, this is not specific to openj9 Docker image, but will also be seen when running in test image based from hotspot Dockerfile

DanHeidinga · 2020-01-21T14:20:43Z

Caused by: java.lang.RuntimeException: librocksdbjni-linuxs390x.so was not found inside JAR.

This is the key thing - the JNI natives for the librocksdbjni library aren't available on s390.

How critical is this component to Quarkus? Is it used by the core framework or only by test cases?

gsmet · 2020-01-22T09:19:42Z

@gunnarmorling could you answer the question above ^? Thanks!

gunnarmorling · 2020-01-22T10:54:19Z

It's used by the Quarkus Kafka Streams extension (or rather, by Kafka Streams itself, which uses RocksDB as a state store). Best for now IMO would be to modify the extension so it treats RocksDB as an optional dependency, i.e. absence of those native libs would be gracefully handled and Kafka Streams could be used on Z without state stores (I hope that's doable). Limiting its capabilities of course, but better than nothing. It also might be another motivator to explore using Infinispan as an alternative state store for Kafka Streams.

wfrank2509 · 2020-02-07T12:21:40Z

The issue seems not related to s390, only.

The quarkus-quickstarts/kafka-streams example throws the following exception when staring the docker container with the native build:

Exception in thread "main" java.lang.RuntimeException: Failed to start quarkus
        at io.quarkus.runner.ApplicationImpl.doStart(ApplicationImpl.zig:202)
2020-02-07 09:26:02,744 ERROR [io.qua.application] (main) Failed to start application: java.lang.ExceptionInInitializerError
        at io.quarkus.runtime.Application.start(Application.java:87)
        at com.oracle.svm.core.hub.ClassInitializationInfo.initialize(ClassInitializationInfo.java:290)
        at io.quarkus.runtime.Application.run(Application.java:210)
        at io.quarkus.runner.GeneratedMain.main(GeneratedMain.zig:41)
Caused by: java.lang.ExceptionInInitializerError
        at com.oracle.svm.core.hub.ClassInitializationInfo.initialize(ClassInitializationInfo.java:290)
        at java.lang.Class.ensureInitialized(DynamicHub.java:475)
        at io.quarkus.kafka.streams.runtime.KafkaStreamsRecorder.loadRocksDb(KafkaStreamsRecorder.java:15)
        at io.quarkus.deployment.steps.KafkaStreamsProcessor$configureAndLoadRocksDb72.deploy_0(KafkaStreamsProcessor$configureAndLoadRocksDb72.zig:65)
        at io.quarkus.deployment.steps.KafkaStreamsProcessor$configureAndLoadRocksDb72.deploy(KafkaStreamsProcessor$configureAndLoadRocksDb72.zig:36)
        at io.quarkus.runner.ApplicationImpl.doStart(ApplicationImpl.zig:76)
        at java.lang.Class.ensureInitialized(DynamicHub.java:475)
        ... 3 more
        at io.quarkus.kafka.streams.runtime.KafkaStreamsRecorder.loadRocksDb(KafkaStreamsRecorder.java:15)
Caused by: java.lang.RuntimeException: librocksdbjni-linux64.so was not found inside JAR.
        at io.quarkus.deployment.steps.KafkaStreamsProcessor$configureAndLoadRocksDb72.deploy_0(KafkaStreamsProcessor$configureAndLoadRocksDb72.zig:65)
        at io.quarkus.deployment.steps.KafkaStreamsProcessor$configureAndLoadRocksDb72.deploy(KafkaStreamsProcessor$configureAndLoadRocksDb72.zig:36)
        at io.quarkus.runner.ApplicationImpl.doStart(ApplicationImpl.zig:76)
        at io.quarkus.runtime.Application.start(Application.java:87)
        at io.quarkus.runtime.Application.run(Application.java:210)
        at io.quarkus.runner.GeneratedMain.main(GeneratedMain.zig:41)
Caused by: java.lang.RuntimeException: librocksdbjni-linux64.so was not found inside JAR.
        at org.rocksdb.NativeLibraryLoader.loadLibraryFromJarToTemp(NativeLibraryLoader.java:110)
        at org.rocksdb.NativeLibraryLoader.loadLibraryFromJarToTemp(NativeLibraryLoader.java:110)
        at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)
        at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)
        at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)
        at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)
        at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:64)
        at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:64)
        at org.rocksdb.RocksDB.<clinit>(RocksDB.java:35)
        at org.rocksdb.RocksDB.<clinit>(RocksDB.java:35)
        at com.oracle.svm.core.hub.ClassInitializationInfo.invokeClassInitializer(ClassInitializationInfo.java:350)
        at com.oracle.svm.core.hub.ClassInitializationInfo.invokeClassInitializer(ClassInitializationInfo.java:350)
        at com.oracle.svm.core.hub.ClassInitializationInfo.initialize(ClassInitializationInfo.java:270)
        at com.oracle.svm.core.hub.ClassInitializationInfo.initialize(ClassInitializationInfo.java:270)
        ... 8 more
        ... 8 more

The build was done with Graalvm 19.3:

./mvnw clean package -Pnative -Dnative-image.container-runtime=docker -Dquarkus.native.builder-image="quay.io/quarkus/ubi-quarkus-native-image:19.3.1-java8"

Same issue occurs with 19.2 and also with building everything with java 11 and using java11 native-image.

One strange observation is that the bundled RocksDB jar ('/Users/wfrank/quarkus-quickstarts/kafka-streams-quickstart/aggregator/target/kafka-streams-quickstart-aggregator-1.0-SNAPSHOT-native-image-source-jar/lib') is not deflatable on macOS by some zip tools but just in the terminal using unzip. There the corresponding *.so files are included.

Any idea under which rock I can look next to investigate further?

gunnarmorling · 2020-02-07T12:26:07Z

Could you try with the latest snapshot build or by building Quarkus and this extension from master? It might be you're running into an issue which has been recently fixed by PR #6952.

wfrank2509 · 2020-02-07T18:13:37Z

Tried the latest SNAPSHOT now.

That fixes the inclusion of the shared library but leads to the following issue when running the aggregator container:

2020-02-07 18:07:54,202 INFO  [org.apa.kaf.str.KafkaStreams] (pool-1-thread-1) stream-client [temperature-aggregator-556bbb04-1780-440c-99d1-b77d486b49a2] State transition from CREATED to REBALANCING
RocksDBExceptionJni::ThrowNew/class - Error: unexpected exception!
Exception in thread "temperature-aggregator-556bbb04-1780-440c-99d1-b77d486b49a2-GlobalStreamThread" java.lang.NoClassDefFoundError: Lorg/rocksdb/RocksDBException;
        at com.oracle.svm.jni.functions.JNIFunctions.FindClass(JNIFunctions.java:317)
        at org.rocksdb.RocksDB.open(RocksDB.java)
        at org.rocksdb.RocksDB.open(RocksDB.java:286)
        at org.apache.kafka.streams.state.internals.RocksDBTimestampedStore.openRocksDB(RocksDBTimestampedStore.java:72)
        at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:180)
        at org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:202)
        at org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
        at org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:58)
        at org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
        at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.lambda$init$0(MeteredKeyValueStore.java:107)
        at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.measureLatency(MeteredKeyValueStore.java:260)
        at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:105)
        at org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.initialize(GlobalStateManagerImpl.java:136)
        at org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.initialize(GlobalStateUpdateTask.java:61)
        at org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.initialize(GlobalStreamThread.java:229)
        at org.apache.kafka.streams.processor.internals.GlobalStreamThread.initialize(GlobalStreamThread.java:345)
        at org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:270)
        at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:497)
        at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)

That leads to the issue my other colleague Andreas has :)

Please see: #7066

gunnarmorling · 2020-02-08T09:15:37Z

Ok, best to not conflate multiple issues; this one here really is about the lack of RocksDB support on s390; this will be hard to come by, unless the native RocksDB parts get ported or we add an alternative statestore implementation that's purely Java-based.

The similar issue about the native lib not being found in a container build for Linux has been addressed already. Let's continue the discussion of the NCDFE over at #7066.

geoand · 2024-09-30T10:23:58Z

Is this still an issue?

smlambert added the kind/bug Something isn't working label Jan 21, 2020

smlambert mentioned this issue Jan 21, 2020

Ensure Quarkus in JVM-mode works on Eclipse OpenJ9 #6538

Closed

7 tasks

wfrank2509 mentioned this issue Feb 7, 2020

Native build of Kafka Streams fails at runtime with class RocksDBException not found #7066

Closed

gunnarmorling added the area/kafka-streams label Feb 8, 2020

geoand added the triage/needs-feedback We are waiting for feedback. label Sep 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

io.quarkus.it.kafka.streams.KafkaStreamsTest.testKafkaStreams fail to locate JNI library on Z #6667

io.quarkus.it.kafka.streams.KafkaStreamsTest.testKafkaStreams fail to locate JNI library on Z #6667

smlambert commented Jan 21, 2020

DanHeidinga commented Jan 21, 2020

gsmet commented Jan 22, 2020

gunnarmorling commented Jan 22, 2020 •

edited

Loading

wfrank2509 commented Feb 7, 2020

gunnarmorling commented Feb 7, 2020 via email

wfrank2509 commented Feb 7, 2020 •

edited

Loading

gunnarmorling commented Feb 8, 2020

geoand commented Sep 30, 2024

io.quarkus.it.kafka.streams.KafkaStreamsTest.testKafkaStreams fail to locate JNI library on Z #6667

io.quarkus.it.kafka.streams.KafkaStreamsTest.testKafkaStreams fail to locate JNI library on Z #6667

Comments

smlambert commented Jan 21, 2020

DanHeidinga commented Jan 21, 2020

gsmet commented Jan 22, 2020

gunnarmorling commented Jan 22, 2020 • edited Loading

wfrank2509 commented Feb 7, 2020

gunnarmorling commented Feb 7, 2020 via email

wfrank2509 commented Feb 7, 2020 • edited Loading

gunnarmorling commented Feb 8, 2020

geoand commented Sep 30, 2024

gunnarmorling commented Jan 22, 2020 •

edited

Loading

wfrank2509 commented Feb 7, 2020 •

edited

Loading