Make ShardingSphere Proxy in GraalVM Native Image form available

[linghengqian]

Feature Request

For English only, other languages will not accept.

Please pay attention on issues you submitted, because we maybe need more details.
If no response anymore and we cannot make decision by current information, we will close it.

Please answer these questions before submitting your issue. Thanks!

Is your feature request related to a problem?

• Yes. Refer to Build GraalVM Native Image nightly for ShardingSphere Proxy #21051 and Graalvm native image cause with Groovy 4 #17779 .

Describe the feature you would like.

• There is currently a nightly build of ShardingSphere Proxy at https://github.com/orgs/apache/packages/container/package/shardingsphere-proxy-native. But it didn't reach usable status because it wasn't ready yet. For some related error logs, please refer to the logs of https://github.com/apache/shardingsphere/actions/runs/3185102779/jobs/5194279014 .

• This issue aims to do the following to bring the nightly build to a usable state.

• Add nightly builds of GitHub Action Job and corresponding documentation.

• Introduces JUnit Platform and JUnit Vintage to replace JUnit 4 to provide the ability to unit test inside GraalVM Native Image. Reference Support JUnit 4 via the VintageTestEngine graalvm/native-build-tools#53 . This issue may need to wait for Migrate certain aspects of JUnit support to the JUnit 5 project graalvm/native-build-tools#313 to be closed before it is actually resolved.

• Update most of the dependencies to match the metadata provided by https://github.com/oracle/graalvm-reachability-metadata , including but not limited to com.zaxxer:HikariCP:3.4.2. This also involves submitting metadata for new third-party dependencies to https://github.com/oracle/graalvm-reachability-metadata . Of course, this problem may also be alleviated by Introduce a default-for attribute oracle/graalvm-reachability-metadata#62 .

• Introduce GraalVM Agent in GraalVM Native Build Tools. I hope to wait for Add agent modes to Maven plugin graalvm/native-build-tools#260 to be resolved. But if the problem mentioned in Maven agent modes support work graalvm/native-build-tools#315 is still not solved, I will follow what I mentioned in https://twitter.com/linghengqian/status/1575865141198934016?s=20&t=iVsRyWdOhmXeAMt_bnXRyg operation to introduce it.

• Update the GraalVM CE version used by GitHub Actions to 22.3.0 or higher to resolve an issue involving server.yaml in ShardingSphere Proxy not being scanned from the classpath. Refer to Cannot resolve FileSystem root resource in Native Image oracle/graal#5020 and Native Image FileSystem is trailing slash sensitive oracle/graal#5080 .

• Works with Groovy 4 specific issues. Its GraalVM Reachability Metadata seems very special that we shouldn't commit metadata to Oracle's Git repository. It's metadata looks like what is mentioned in Update Groovy version to fix the problem about IndyInterface #18175 and Can't build Groovy Application graalvm/native-build-tools#324 (comment) dynamically.

• Working on a fix for Apache Calcite, due to [CALCITE-4882] Introduce new Lambda-based Metadata framework calcite#2603 being closed, may need to be introduced in a unique way https://twitter.com/intjesus/status/1567276748257259520?s=20&t=nUlN7LI8e9_2KO9lPwCqaw, Implement Calcite native library that parses sql and returns a binary substrait representation substrait-io/substrait#21 and a component of https://github.com/substrait-io/substrait-java/tree/main/isthmus . This allowed us to finally get rid of https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/util/javac/JaninoCompiler.java like this, Janino related classes .

• The PRs involved in this issue are as follows:

  • Update Mockito and ByteBuddy to help with GraalVM integration #17665
  • Update Groovy version to fix the problem about IndyInterface #18175
  • Update Apache Calcite to 1.30.0 to improve GraalVM support #19737
  • Update H2DataBase to 2.x and drop support for H2Database 1.x #19964
  • Update Junit to 4.13.2 and migrate deprecated APIs to Hamcrest #20937
  • Update CODE_OF_CONDUCT about assertThat #21046
  • Build GraalVM Native Image nightly for ShardingSphere Proxy #21109
  • Fixes documentation for building GraalVM Native Image #21341
  • Update library to serve building GraalVM Native Image #21571
  • Add Release Profile for distribution/proxy-native #21657
  • Fix nightly builds of GraalVM Native Image #21688
  • Upgrade dependent versions and fix plugin settings for building GraalVM Native Image #22169
  • Add support for com.github.luben:zstd-jni:1.5.2-5 oracle/graalvm-reachability-metadata#121
  • Add support for java.sql.Date in IntervalShardingAlgorithm #22495
  • Add support for com.google.protobuf:protobuf-java-util:3.21.12 oracle/graalvm-reachability-metadata#167
  • Add support for org.opengauss:opengauss-jdbc:3.1.0-og oracle/graalvm-reachability-metadata#168
  • Add support for com.hazelcast:hazelcast:5.2.1 oracle/graalvm-reachability-metadata#169
  • Add support for io.etcd:jetcd-core:0.7.5 oracle/graalvm-reachability-metadata#170
  • Add user-code-filter.json for unit tests of org.opengauss:opengauss-jdbc oracle/graalvm-reachability-metadata#174
  • Add support for javax.cache:cache-api:1.1.1 oracle/graalvm-reachability-metadata#176
  • Add support for com.github.ben-manes.caffeine:caffeine:3.1.2 oracle/graalvm-reachability-metadata#193
  • Use GraalVM CE 22.3.1 to fix Nightly Build #23728
  • Remove when entries for mockito that are not used in unit tests shardingsphere-elasticjob#2179
  • Introduce Truffle Espresso to make GroovyShell available under GraalVM Native Image #23873
  • Fix nightly builds and release profile of ShardingSphere Proxy Native #23957
  • Update the project's Junit and GraalVM Native Build Tools versions oracle/graalvm-reachability-metadata#218
  • Update seata-all to handle cglib breaking unit test execution with GraalVM Tracing Agent #24177
  • Initialize the generateStandardMetadata Maven Profile of all module #24271
  • Fix wrong classpath in Espresso context #24912
  • Update seata-all and netty to fix nativeTest in GraalVM Native Build Tools #24929
  • Supports compilation through OpenJDK 20 and its downstream distributions #25181
  • Re-design for ShardingSphere groovy expr #25509
  • Restrict Espresso API to only detect JAVA_HOME under Native Image #25515
  • Remove deleted module from the infra-util module POM #25523
  • Fix build-proxy-native-image job on Nightly Build #25632
  • Update the version of GraalVM CE used by ShardingSphere Proxy Native to 23.0.1 #27487
  • Relax time limits for nightly builds involving test with GraalVM Tracing Agent #28060
  • Disable the implementation of Etcd during the build of ShardingSphere Proxy Native #28194
  • Fixes OutOfMemoryError unique to building GraalVM Native Image on Github Actions devices #28213
  • Refactor shardingsphere-infra-expr to expose the use of Row Value Expressions SPI #28340
  • Add support for org.apache.commons:commons-dbcp2:2.9.0 oracle/graalvm-reachability-metadata#380
  • Add support for com.github.ben-manes.caffeine:caffeine:2.9.3 oracle/graalvm-reachability-metadata#388
  • Support org.junit.jupiter.api.condition.EnabledOnOs in NativeTest graalvm/native-build-tools#504
  • Add support for org.apache.shardingsphere.elasticjob:elasticjob-lite-core:3.0.3 oracle/graalvm-reachability-metadata#390
  • Support for building with OpenJDK 21 #28326
  • Update the infra-expr-espresso module related GraalVM CE version to 23.1.1 #28609
  • Add documentation about the infra-expr-espresso module #28610
  • Explicitly declare dependencies for plexus-xml and plexus-utils graalvm/native-build-tools#484
  • Support org.junit.jupiter.api.condition.EnabledOnOs in NativeTest graalvm/native-build-tools#504
  • Provides built-in GraalVM Reachability Metadata for core features of ShardingSphere JDBC and adds nativeTest-related unit test subsets #29000
  • Open the resource:/ related FileSystem for JDBCRepositorySQLLoader in the GraalVM Native Image environment #29009
  • Migrate JAXB Runtime from com.sun.xml.bind:jaxb-impl to org.glassfish.jaxb:jaxb-runtime #29013
  • Make JDBCRepositorySQLLoader avoid using Path#toFile in GraalVM Native Image #29014
  • Declare in the document that ReadWriteSplitting Config is parsed by GroovyShell by default and adds corresponding unit tests #29034

[linghengqian]

• I am working on this issue. However, this issue is affected by many issues and is unlikely to be resolved in the short term.

[linghengqian]

• I'm assuming the GraalVM reachability metadata about Apache Curator will be migrated out of Spring Cloud since their community just migrated the corresponding metadata for Consul Client 1.4.5.

[linghengqian]

• I did intentionally bump the version of Vertx used by the master branch of ShardingSphere from 4.3.3 to 4.3.4 to be faster to accept issues like Support GraalVM Reachability Metadata to Simplify Native Compilation vert-x3/issues#604. But vertx-sql-client:4.3.4 seems to have refactorings that https://vertx.io/blog/eclipse-vert-x-4-3-4/ does not directly document, such as around a series of changes to io.vertx.mysqlclient.impl.protocol.ColumnDefinition. I gave up on this idea for now.

• 

• 

• For jetcd-core and atomikos, I explained it at Update library to serve building GraalVM Native Image #21552.

• native-image support with Graal and native-image.properties google/guava#3894 may have an impact on this issue, but I don't think it has any impact, it's just a tool library.

[linghengqian]

• For jetcd-core and atomikos, I explained it at Update library to serve building GraalVM Native Image #21552.

• I found and resolved the conflict around astomikos ,Jetcd and vertx in the new PR, refer to Update library to serve building GraalVM Native Image #21571 . There were some breaking changes in hikaricp 4.0.3, so I've temporarily removed tweaks to it.
• I don't feel the need to switch to Junit Jupiter right away, so I still use Junit Vintage for the relevant PRs.
• 
• The new properties of HikarciCP are blocking my PR and I feel like I need to handle it better in a separate PR. Because it also involves how to match to HikariCP 5.0.1's GraalVM reachability metadata.
• 

[linghengqian]

• I will be adding a release profile to distribution/proxy-native to help with its release in ShardingSphere 5.2.2 and later.

• update in 2022.10.19: Maybe one day in the future, we no longer need to limit the JVM that generates GraalVM Native Inage to GraalVM. Refer to JavaOne 2022.

• I saw some additional issues when I checked the GHA logs, maybe something needs to be done for BouncyCastle ALPN seems to break native image builds netty/netty#11369 which has not been closed.

• In addition, for the specific exception exposed when using GraalVM Native Image, we need to wait for the closure of [GR-40106] Reporting missing metadata in Native Image by throwing special exceptions oracle/graal#5171 . For some unreasonable warnings when using GraalVM reachability metadata, it may be necessary to wait for the closure of [GR-49770] Safe Composition of Metadata  oracle/graal#5173 .

[linghengqian]

• Remove the restriction that the Authority Rule must be configured #20457 and Fixes #20457, allows to start proxy without configuring authority rule. #21599 finally allow ShardingSphere Proxy to be started without a separate server.yaml, which will be extremely and greatly simplifies the use of GraalVM Native Image. But it still seems that the mode configuration cannot be added through DistSQL? 🤔
• update: It looks like the PR has been merged, but I still need some unit test validation locally, and it looks like the documentation for building GraalVM Native Image shouldn't be updated with information related to this right away. Some filesystem fixes came to GraalVM's master branch, but it doesn't seem to make it into dev-build yet.

[linghengqian]

• The Log at https://github.com/apache/shardingsphere/actions/runs/3298856474/jobs/5441511861 shows that the CI of GraalVM Native Image is broken. This question first came from Refactor : integrate the different build job together #21677 . I will submit a PR to fix it.

[linghengqian]

• I saw some incorrectly imported metadata at https://github.com/apache/shardingsphere/actions/runs/3303870293/jobs/5452309095 . Looks like most of it is related to Netty. I'm not quite sure what I need to do, considering that the Spring community actually has this problem. Reference https://twitter.com/sdeleuze/status/1583493681386225664?t=2IBnsStmiaTmiqU_BqwpMw&s=19 .

• It looks like Add support for Log4j2 oracle/graalvm-reachability-metadata#50 explains the issues related to Log4j2. Pending https://issues.apache.org/jira/browse/LOG4J2-2649 . This seems to require us to force-pull the Log4j2 version in the native profile to 3.0.0-SNAPSHOT. Should I say using SNAPSHOT is a weird behavior?

• It just looks like log4j2's 3.0.0-SNAPSHOT completely removes stuff related to java.lang.invoke.LambdaMetafactory, the graalvm team almost doesn't want to support such usage at all, which involves SecurityManager. I assume that I should be able to find a simple operation in the native profile of the shardingsphere to specify the version of all the passed log4j2 dependencies to 3.0.0-SNAPSHOT, and also remove the log4j1 from the upstream dependencies, and finally Add support for Log4j2 oracle/graalvm-reachability-metadata#50 can be reopened and the metadata of log4j2 can be added. However, I am quite depressed how ShardingSphere CI Log can see a hamcrest related class when building GraalVM Native Image warning, the test library is not in the final product.

• I saw some related issues at Support for graal native-image luben/zstd-jni#115 that might be related to the class below.

• The classes of the metadata involved are as follows.

com.github.luben.zstd.ZstdCompressCtx
org.conscrypt.ConscryptEngine
ch.qos.logback.core.encoder.LayoutWrappingEncoder
com.aayushatharva.brotli4j.Brotli4jLoader
com.github.luben.zstd.Zstd
com.ning.compress.lzf.impl.UnsafeChunkDecoder
io.netty.handler.codec.compression.Lz4FrameDecoder
net.jpountz.lz4.LZ4Exception
io.netty.handler.codec.compression.Lz4FrameEncoder
io.netty.handler.codec.marshalling.CompatibleMarshallingDecoder
org.jboss.marshalling.ByteInput
io.netty.handler.codec.marshalling.CompatibleMarshallingEncoder
org.jboss.marshalling.ByteOutput
io.netty.handler.codec.marshalling.MarshallingDecoder
io.netty.handler.codec.marshalling.MarshallingEncoder
org.apache.commons.logging.impl.Log4JLogger
org.apache.log4j.Category
org.apache.log4j.Level
org.apache.log4j.Priority
org.apache.logging.log4j.Logger
org.apache.logging.log4j.core.LoggerContext
org.apache.logging.log4j.message.DefaultFlowMessageFactory
org.apache.logging.log4j.message.ReusableMessageFactory
org.conscrypt.Conscrypt
org.conscrypt.OpenSSLContextImpl
org.hamcrest.number.OrderingComparison
com.zaxxer.hikari.HikariConfig

[linghengqian]

• It looks like Add support for Log4j2 oracle/graalvm-reachability-metadata#50 explains the issues related to Log4j2. Pending https://issues.apache.org/jira/browse/LOG4J2-2649 . This seems to require us to force-pull the Log4j2 version in the native profile to 3.0.0-SNAPSHOT. Should I say using SNAPSHOT is a weird behavior?

• It just looks like log4j2's 3.0.0-SNAPSHOT completely removes stuff related to java.lang.invoke.LambdaMetafactory, the graalvm team almost doesn't want to support such usage at all, which involves SecurityManager. I assume that I should be able to find a simple operation in the native profile of the shardingsphere to specify the version of all the passed log4j2 dependencies to 3.0.0-SNAPSHOT, and also remove the log4j1 from the upstream dependencies, and finally Add support for Log4j2 oracle/graalvm-reachability-metadata#50 can be reopened and the metadata of log4j2 can be added. However, I am quite depressed how ShardingSphere CI Log can see a hamcrest related class when building GraalVM Native Image warning, the test library is not in the final product.

• Hopefully Remove Log4j2 entries from netty-common's reflection metadata oracle/graalvm-reachability-metadata#108 will remove the warning related to Log4j2.

[linghengqian]

• In https://github.com/apache/shardingsphere/actions/runs/3481017089/jobs/5821561497 , the results displayed in the log are still not optimistic. I'm doing additional investigation locally.

Warning: Could not resolve class com.github.luben.zstd.ZstdCompressCtx for reflection configuration. Reason: java.lang.ClassNotFoundException: com.github.luben.zstd.ZstdCompressCtx.
Warning: Could not resolve class org.conscrypt.ConscryptEngine for reflection configuration. Reason: java.lang.ClassNotFoundException: org.conscrypt.ConscryptEngine.
Warning: Could not resolve class org.conscrypt.ConscryptEngine for reflection configuration. Reason: java.lang.ClassNotFoundException: org.conscrypt.ConscryptEngine.
Warning: Could not resolve class org.conscrypt.ConscryptEngine for reflection configuration. Reason: java.lang.ClassNotFoundException: org.conscrypt.ConscryptEngine.
Warning: Method ch.qos.logback.core.encoder.LayoutWrappingEncoder.setParent(Appender) not found.
Warning: Could not resolve class com.aayushatharva.brotli4j.Brotli4jLoader for reflection configuration. Reason: java.lang.ClassNotFoundException: com.aayushatharva.brotli4j.Brotli4jLoader.
Warning: Could not resolve class com.github.luben.zstd.Zstd for reflection configuration. Reason: java.lang.ClassNotFoundException: com.github.luben.zstd.Zstd.
Warning: Could not resolve class com.ning.compress.lzf.impl.UnsafeChunkDecoder for reflection configuration. Reason: java.lang.ClassNotFoundException: com.ning.compress.lzf.impl.UnsafeChunkDecoder.
Warning: Could not register io.netty.handler.codec.compression.Lz4FrameDecoder: queryAllPublicMethods for reflection. Reason: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Exception.
Warning: Could not register io.netty.handler.codec.compression.Lz4FrameEncoder: queryAllPublicMethods for reflection. Reason: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Exception.
Warning: Could not register io.netty.handler.codec.marshalling.CompatibleMarshallingDecoder: queryAllPublicMethods for reflection. Reason: java.lang.NoClassDefFoundError: org/jboss/marshalling/ByteInput.
Warning: Could not register io.netty.handler.codec.marshalling.CompatibleMarshallingEncoder: queryAllPublicMethods for reflection. Reason: java.lang.NoClassDefFoundError: org/jboss/marshalling/ByteOutput.
Warning: Could not register io.netty.handler.codec.marshalling.MarshallingDecoder: queryAllPublicMethods for reflection. Reason: java.lang.NoClassDefFoundError: org/jboss/marshalling/ByteInput.
Warning: Could not register io.netty.handler.codec.marshalling.MarshallingEncoder: queryAllPublicMethods for reflection. Reason: java.lang.NoClassDefFoundError: org/jboss/marshalling/ByteOutput.
Warning: Could not register method org.apache.commons.logging.impl.Log4JLogger.<init>(String) for reflection. Reason: java.lang.NoClassDefFoundError: org/apache/log4j/Category.
Warning: Could not resolve class org.apache.log4j.Level for reflection configuration. Reason: java.lang.ClassNotFoundException: org.apache.log4j.Level.
Warning: Could not resolve class org.apache.log4j.Priority for reflection configuration. Reason: java.lang.ClassNotFoundException: org.apache.log4j.Priority.
Warning: Could not resolve class org.apache.logging.log4j.Logger for reflection configuration. Reason: java.lang.ClassNotFoundException: org.apache.logging.log4j.Logger.
Warning: Could not resolve class org.apache.logging.log4j.core.LoggerContext for reflection configuration. Reason: java.lang.ClassNotFoundException: org.apache.logging.log4j.core.LoggerContext.
Warning: Could not resolve class org.apache.logging.log4j.message.DefaultFlowMessageFactory for reflection configuration. Reason: java.lang.ClassNotFoundException: org.apache.logging.log4j.message.DefaultFlowMessageFactory.
Warning: Could not resolve class org.apache.logging.log4j.message.ReusableMessageFactory for reflection configuration. Reason: java.lang.ClassNotFoundException: org.apache.logging.log4j.message.ReusableMessageFactory.
Warning: Could not resolve class org.conscrypt.Conscrypt for reflection configuration. Reason: java.lang.ClassNotFoundException: org.conscrypt.Conscrypt.
Warning: Could not resolve class org.conscrypt.OpenSSLContextImpl$TLSv13 for reflection configuration. Reason: java.lang.ClassNotFoundException: org.conscrypt.OpenSSLContextImpl$TLSv13.
Warning: Could not resolve class org.hamcrest.number.OrderingComparison for reflection configuration. Reason: java.lang.ClassNotFoundException: org.hamcrest.number.OrderingComparison.
Warning: Method com.zaxxer.hikari.HikariConfig.getScheduledExecutorService() not found.
Warning: Method com.zaxxer.hikari.HikariConfig.isInitializationFailFast() not found.
Warning: Method com.zaxxer.hikari.HikariConfig.isJdbc4ConnectionTest() not found.
Warning: Method com.zaxxer.hikari.HikariConfig.setInitializationFailFast(boolean) not found.
Warning: Method com.zaxxer.hikari.HikariConfig.setJdbc4ConnectionTest(boolean) not found.
Warning: Method com.zaxxer.hikari.HikariConfig.setScheduledExecutorService(ScheduledThreadPoolExecutor) not found.
Warning: Could not register io.netty.handler.codec.compression.Lz4FrameDecoder: queryAllPublicMethods for reflection. Reason: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Exception.
Warning: Could not register io.netty.handler.codec.compression.Lz4FrameEncoder: queryAllPublicMethods for reflection. Reason: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Exception.
Warning: Could not register io.netty.handler.codec.marshalling.CompatibleMarshallingDecoder: queryAllPublicMethods for reflection. Reason: java.lang.NoClassDefFoundError: org/jboss/marshalling/ByteInput.
Warning: Could not register io.netty.handler.codec.marshalling.CompatibleMarshallingEncoder: queryAllPublicMethods for reflection. Reason: java.lang.NoClassDefFoundError: org/jboss/marshalling/ByteOutput.
Warning: Could not register io.netty.handler.codec.marshalling.MarshallingDecoder: queryAllPublicMethods for reflection. Reason: java.lang.NoClassDefFoundError: org/jboss/marshalling/ByteInput.
Warning: Could not register io.netty.handler.codec.marshalling.MarshallingEncoder: queryAllPublicMethods for reflection. Reason: java.lang.NoClassDefFoundError: org/jboss/marshalling/ByteOutput.

[linghengqian]

• When ShardingSphere tests the building of GraalVM Native Image, it is found that io.vertx.core.impl.AddressResolver lacks GraalVM Reachability metadata. This class belongs to io.vertx:vertx-core:4.3.3. Therefore, it is linked to the new issue. For example BouncyCastle ALPN seems to break native image builds netty/netty#11369 .
• 

[linghengqian]

• I initialized the first part of GRM for shardingsphere-jdbc-core at Some plan to write additional unit tests for the class library need GraalVM Reachability Metadata linghengqian/graalvm-trace-metadata-smoketest#1. But I saw Update IntervalShardingAlgorithm.java to fix an exception caused by polymorphism #20255 related issue. I will submit a PR for this. I also opened [GR-43010] Using Groovy classes under native-image results in UnsupportedFeatureError oracle/graal#5522.

[linghengqian]

• Once the Remove Vert.x from ShardingSphere-Proxy #22982 is merged, nativeTest with vertx error log will no longer exist.

[linghengqian]

• At [GR-46399] [GR-48232] [GR-48207] [GR-48180] Various important bug fixes for Native Image. oracle/graal#7277, the memory usage behavior of GraalVM CE Dev 23.1.0 for JDK21 has changed again. After that one should consider rolling back the changes made by Fixes OutOfMemoryError unique to building GraalVM Native Image on Github Actions devices #28213.

• At the same time, we still need to wait for the release of Lombok to solve ShardingSphere fails to build on OpenJDK 21 EA #28293.

• The URL used by the documentation should be changed, as the GraalVM Team provided a new documentation URL Tag a while ago at External link is broken in Native Build Tips documentation page quarkusio/quarkus#34929 .

• Once the issue involved in Refactor shardingsphere-infra-expr to expose the use of Row Value Expressions SPI #28340 is resolved, we will hopefully open nativeTest on the ShardingSphere side, although the support for multi-module projects in the Maven Plugin of Native Build Tools 0.9.25 still exists Visible problems.

[RaigorJiang]

Great work 👍

[zhfeng]

Thanks @linghengqian and it is really very nice work! Do you have any plan to write a simple application to compare the performance between the GraalVM Native and JVM? I also write a quarkus extension to run with shardingsphere-jdbc in native mode which compile and evaluate the groovy rules at the build time.

[linghengqian]

Thanks @linghengqian and it is really very nice work! Do you have any plan to write a simple application to compare the performance between the GraalVM Native and JVM? I also write a quarkus extension to run with shardingsphere-jdbc in native mode which compile and evaluate the groovy rules at the build time.

• Sorry, it is not convenient for me to write a unit test for performance comparison because I lack the knowledge related to stress testing ShardingSphere.

• For groovyshell, the unit tests related to nativeTest have skipped the use of groovyshell. Please refer to https://shardingsphere.apache.org/document/current/cn/user-manual/shardingsphere-jdbc/graalvm-native-image/ . The relevant unit tests are located at org.apache.shardingsphere:shardingsphere-infra-nativetest.

• This involves the buildArgs of --report-unsupported-elements-at-runtim which downstream consumers must use.

• To facilitate review, I compressed the size of the initial subset of unit tests. What can be verified now is that in the standalone mode of ShardingSphere, the five functions and derivative functions of sharding, readwrite-splitting, mask, shadow and encrypt can be used under native image. For cluster mode and DistSQL, I have not added corresponding verification yet, and additional GraalVM Reachability Metadata json files should be required.

• Since the modifications to these json files are not actually related to the current issue, I will discuss them in a new issue later.

• The core problem solved by this issue is that with the help of the new Row Value Expression SPI, you can configure ShardingSphere without using Groovy syntax at all through the buildArgs of --report-unsupported-elements-at-runtim, which makes building a GraalVM Native Image that completes nativeTest becomes a reality.

• If needed, I can help investigate the nativeTest failure issue on the quarkus-shardingsphere side, but this requires quarkus-shardingsphere to synchronize the master branch of shardingsphere. In addition, the naming of related submodules may still be variable in shardingsphere.

[zhfeng]

Well, quarkus-shardingsphere does not depend on GraalVM Reachability Metadata directly but I think it could be helpful to build native images since native-image tool can detect the .json configuartion in the jars and add them automatically. Also I think we need to avoid using --report-unsupported-elements-at-runtim because it can lead to any unexpected error at runtime.

[linghengqian]

Well, quarkus-shardingsphere does not depend on GraalVM Reachability Metadata directly but I think it could be helpful to build native images since native-image tool can detect the .json configuartion in the jars and add them automatically. Also I think we need to avoid using --report-unsupported-elements-at-runtim because it can lead to any unexpected error at runtime.

• This is natural. The current need to initialize GroovyShell related classes is because the default implementation of Row Value
  Expression SPI uses GroovyShell, which results in the need for downstream to actively add buildArgs for compatibility. Maybe I can modify shardingsphere-infra-expr-core to expose a possible configuration item on the ShardingSphere side. Or I can break this non-existent compatibility under GraalVM Native Image.

[zhfeng]

OK, thank for clarifying.

[linghengqian]

Well, quarkus-shardingsphere does not depend on GraalVM Reachability Metadata directly but I think it could be helpful to build native images since native-image tool can detect the .json configuartion in the jars and add them automatically. Also I think we need to avoid using --report-unsupported-elements-at-runtim because it can lead to any unexpected error at runtime.

• @zhfeng I'll try to handle it at Improve GraalVM Reachability Metadata and corresponding nativeTest related unit tests #29052 .