Skip to content

Commit

Permalink
Add GraalVM Reachability Metadata and corresponding nativeTest for Se…
Browse files Browse the repository at this point in the history 
…ata integration
  • Loading branch information
@linghengqian
linghengqian committed Feb 17, 2024
1 parent e91e29d commit 705cd2d
Show file tree
Hide file tree
Showing 18 changed files with 498 additions and 72 deletions.
54 changes: 48 additions & 6 deletions ...ument/content/user-manual/shardingsphere-jdbc/graalvm-native-image/_index.cn.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ plugins {
dependencies {
implementation 'org.apache.shardingsphere:shardingsphere-jdbc-core:${shardingsphere.version}'
implementation 'org.graalvm.buildtools:graalvm-reachability-metadata:0.10.0'
}
graalvmNative {
Expand All @@ -89,6 +90,9 @@ graalvmNative {
buildArgs.add('-H:+AddAllCharsets')
}
}
metadataRepository {
enabled.set(false)
}
}
```

Expand Down Expand Up @@ -181,7 +185,7 @@ Image 下使用。
```

2. 对于 `读写分离` 的功能，你需要使用 `行表达式` SPI 的其他实现，以在配置 `logic database name`，`writeDataSourceName` 和 `readDataSourceNames`
时绕开对 GroovyShell 的调用。一个可能的配置是使用 `LITERAL` 的 `行表达式` SPI 的实现。对于 `数据分片` 的功能的 `actualDataNodes` 同理。
时绕开对 GroovyShell 的调用。一个可能的配置是使用 `LITERAL` 的 `行表达式` SPI 的实现。
```yaml
rules:
- !READWRITE_SPLITTING
Expand All @@ -193,6 +197,18 @@ rules:
- <LITERAL>ds_2
```

对于 `数据分片` 的功能的 `actualDataNodes` 同理。

```yaml
- !SHARDING
tables:
t_order:
actualDataNodes: <LITERAL>ds_0.t_order_0, ds_0.t_order_1, ds_1.t_order_0, ds_1.t_order_1
keyGenerateStrategy:
column: order_id
keyGeneratorName: snowflake
```

3. 使用者依然需要在 `src/main/resources/META-INF/native-image` 文件夹或 `src/test/resources/META-INF/native-image` 文件夹配置独立
文件的 GraalVM Reachability Metadata。使用者可通过 GraalVM Native Build Tools 的 GraalVM Tracing Agent 来快速采集 GraalVM
Reachability Metadata。
Expand All @@ -201,18 +217,44 @@ Reachability Metadata。
当遇到如下 Error，使用者需要添加 `-H:+AddAllCharsets` 的 `buildArg` 到 GraalVM Native Build Tools 的配置中。

```shell
Caused by: java.io.UnsupportedEncodingException: SQL Server collation SQL_Latin1_General_CP1_CI_AS is not supported by this driver.
com.microsoft.sqlserver.jdbc.SQLCollation.encodingFromSortId(SQLCollation.java:506)
com.microsoft.sqlserver.jdbc.SQLCollation.<init>(SQLCollation.java:63)
com.microsoft.sqlserver.jdbc.SQLServerConnection.processEnvChange(SQLServerConnection.java:3174)
[...]
Caused by: java.io.UnsupportedEncodingException: Codepage Cp1252 is not supported by the Java environment.
com.microsoft.sqlserver.jdbc.Encoding.checkSupported(SQLCollation.java:572)
com.microsoft.sqlserver.jdbc.SQLCollation$SortOrder.getEncoding(SQLCollation.java:473)
com.microsoft.sqlserver.jdbc.SQLCollation.encodingFromSortId(SQLCollation.java:501)
[...]
```

5. 当使用 Seata 的 BASE 集成时，用户需要使用特定的 `io.seata:seata-all:1.8.0` 版本以避开对 ByteBuddy Java API 的使用，
并排除 `io.seata:seata-all:1.8.0` 中过时的 `org.antlr:antlr4-runtime:4.8` 的 Maven 依赖。可能的配置例子如下，

```xml
<project>
<dependencies>
<dependency>
<groupId>org.apache.shardingsphere</groupId>
<artifactId>shardingsphere-jdbc-core</artifactId>
<version>${shardingsphere.version}</version>
</dependency>
<dependency>
<groupId>org.apache.shardingsphere</groupId>
<artifactId>shardingsphere-transaction-base-seata-at</artifactId>
<version>${shardingsphere.version}</version>
</dependency>
<dependency>
<groupId>io.seata</groupId>
<artifactId>seata-all</artifactId>
<version>1.8.0</version>
<exclusions>
<exclusion>
<groupId>org.antlr</groupId>
<artifactId>antlr4-runtime</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
</project>
```

## 贡献 GraalVM Reachability Metadata

ShardingSphere 对在 GraalVM Native Image 下的可用性的验证，是通过 GraalVM Native Build Tools 的 Maven Plugin 子项目来完成的。
Expand Down
63 changes: 53 additions & 10 deletions ...ument/content/user-manual/shardingsphere-jdbc/graalvm-native-image/_index.en.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -76,11 +76,12 @@ and the documentation of GraalVM Native Build Tools shall prevail.

```groovy
plugins {
id 'org.graalvm.buildtools.native' version '0.10.0'
id 'org.graalvm.buildtools.native' version '0.10.0'
}
dependencies {
implementation 'org.apache.shardingsphere:shardingsphere-jdbc-core:${shardingsphere.version}'
implementation 'org.apache.shardingsphere:shardingsphere-jdbc-core:${shardingsphere.version}'
implementation 'org.graalvm.buildtools:graalvm-reachability-metadata:0.10.0'
}
graalvmNative {
Expand All @@ -92,6 +93,9 @@ graalvmNative {
buildArgs.add('-H:+AddAllCharsets')
}
}
metadataRepository {
enabled.set(false)
}
}
```

Expand Down Expand Up @@ -186,9 +190,8 @@ normally under GraalVM Native Image.
```

2. For the `ReadWrite Splitting` feature, you need to use other implementations of `Row Value Expressions` SPI to configure
`logic database name`, `writeDataSourceName` and `readDataSourceNames` when bypassing calls to GroovyShell. One possible
configuration is to use the `Row Value Expressions` SPI implementation of `LITERAL`. The same applies to `actualDataNodes`
for the `Sharding` feature.
`logic database name`, `writeDataSourceName` and `readDataSourceNames` when bypassing calls to GroovyShell.
One possible configuration is to use the `Row Value Expressions` SPI implementation of `LITERAL`.

```yaml
rules:
Expand All @@ -201,6 +204,18 @@ rules:
- <LITERAL>ds_2
```

The same applies to `actualDataNodes` for the `Sharding` feature.

```yaml
- !SHARDING
tables:
t_order:
actualDataNodes: <LITERAL>ds_0.t_order_0, ds_0.t_order_1, ds_1.t_order_0, ds_1.t_order_1
keyGenerateStrategy:
column: order_id
keyGeneratorName: snowflake
```

3. Users still need to configure GraalVM Reachability Metadata for independent files in the `src/main/resources/META-INF/native-image`
folder or `src/test/resources/META-INF/native-image` folder. Users can quickly collect GraalVM Reachability Metadata through
the GraalVM Tracing Agent of GraalVM Native Build Tools.
Expand All @@ -210,18 +225,46 @@ will dynamically load different character sets based on the encoding used in the
When encountering the following Error, users need to add the `buildArg` of `-H:+AddAllCharsets` to the configuration of GraalVM Native Build Tools.

```shell
Caused by: java.io.UnsupportedEncodingException: SQL Server collation SQL_Latin1_General_CP1_CI_AS is not supported by this driver.
com.microsoft.sqlserver.jdbc.SQLCollation.encodingFromSortId(SQLCollation.java:506)
com.microsoft.sqlserver.jdbc.SQLCollation.<init>(SQLCollation.java:63)
com.microsoft.sqlserver.jdbc.SQLServerConnection.processEnvChange(SQLServerConnection.java:3174)
[...]
Caused by: java.io.UnsupportedEncodingException: Codepage Cp1252 is not supported by the Java environment.
com.microsoft.sqlserver.jdbc.Encoding.checkSupported(SQLCollation.java:572)
com.microsoft.sqlserver.jdbc.SQLCollation$SortOrder.getEncoding(SQLCollation.java:473)
com.microsoft.sqlserver.jdbc.SQLCollation.encodingFromSortId(SQLCollation.java:501)
[...]
```

5. When using Seata's BASE integration,
users need to use a specific `io.seata:seata-all:1.8.0` version to avoid using the ByteBuddy Java API,
and exclude the outdated Maven dependency of `org.antlr:antlr4-runtime:4.8` in `io.seata:seata-all:1.8.0`.
Possible configuration examples are as follows,

```xml
<project>
<dependencies>
<dependency>
<groupId>org.apache.shardingsphere</groupId>
<artifactId>shardingsphere-jdbc-core</artifactId>
<version>${shardingsphere.version}</version>
</dependency>
<dependency>
<groupId>org.apache.shardingsphere</groupId>
<artifactId>shardingsphere-transaction-base-seata-at</artifactId>
<version>${shardingsphere.version}</version>
</dependency>
<dependency>
<groupId>io.seata</groupId>
<artifactId>seata-all</artifactId>
<version>1.8.0</version>
<exclusions>
<exclusion>
<groupId>org.antlr</groupId>
<artifactId>antlr4-runtime</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
</project>
```

## Contribute GraalVM Reachability Metadata

The verification of ShardingSphere's availability under GraalVM Native Image is completed through the Maven Plugin subproject
Expand Down
21 changes: 18 additions & 3 deletions ...es/META-INF/native-image/com.github.ben-manes.caffeine/caffeine/2.9.3/reflect-config.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,15 +40,25 @@
"fields":[{"name":"value"}]
},
{
"condition":{"typeReachable":"com.github.benmanes.caffeine.cache.NodeFactory"},
"name":"com.github.benmanes.caffeine.cache.PDMS",
"condition":{"typeReachable":"com.github.benmanes.caffeine.cache.PDW"},
"name":"com.github.benmanes.caffeine.cache.PDW",
"fields":[{"name":"writeTime"}]
},
{
"condition":{"typeReachable":"com.github.benmanes.caffeine.cache.SIMSW"},
"name":"com.github.benmanes.caffeine.cache.PDWMS",
"methods":[{"name":"<init>","parameterTypes":[] }]
},
{
"condition":{"typeReachable":"com.github.benmanes.caffeine.cache.LocalCacheFactory"},
"condition":{"typeReachable":"com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache"},
"name":"com.github.benmanes.caffeine.cache.SIMS",
"methods":[{"name":"<init>","parameterTypes":["com.github.benmanes.caffeine.cache.Caffeine","com.github.benmanes.caffeine.cache.CacheLoader","boolean"] }]
},
{
"condition":{"typeReachable":"com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache"},
"name":"com.github.benmanes.caffeine.cache.SIMSW",
"methods":[{"name":"<init>","parameterTypes":["com.github.benmanes.caffeine.cache.Caffeine","com.github.benmanes.caffeine.cache.CacheLoader","boolean"] }]
},
{
"condition":{"typeReachable":"com.github.benmanes.caffeine.cache.StripedBuffer"},
"name":"com.github.benmanes.caffeine.cache.StripedBuffer",
Expand All @@ -58,5 +68,10 @@
"condition":{"typeReachable":"com.github.benmanes.caffeine.cache.StripedBuffer"},
"name":"java.lang.Thread",
"fields":[{"name":"threadLocalRandomProbe"}]
},
{
"condition":{"typeReachable":"com.github.benmanes.caffeine.cache.NodeFactory"},
"name":"com.github.benmanes.caffeine.cache.PDMS",
"methods":[{"name":"<init>","parameterTypes":[] }]
}
]
43 changes: 14 additions & 29 deletions .../org.apache.shardingsphere/shardingsphere-infra-reachability-metadata/reflect-config.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -65,31 +65,6 @@
"condition":{"typeReachable":"org.apache.shardingsphere.infra.util.yaml.YamlEngine"},
"name":"org.apache.shardingsphere.driver.api.yaml.YamlJDBCConfigurationCustomizer"
},
{
"condition":{"typeReachable":"org.apache.shardingsphere.driver.jdbc.core.driver.ShardingSphereURLManager"},
"name":"org.apache.shardingsphere.driver.jdbc.core.driver.spi.absolutepath.AbsolutePathURLProvider",
"methods":[{"name":"<init>","parameterTypes":[] }]
},
{
"condition":{"typeReachable":"org.apache.shardingsphere.driver.jdbc.core.driver.ShardingSphereURLManager"},
"name":"org.apache.shardingsphere.driver.jdbc.core.driver.spi.absolutepath.AbsolutePathWithEnvironmentURLProvider",
"methods":[{"name":"<init>","parameterTypes":[] }]
},
{
"condition":{"typeReachable":"org.apache.shardingsphere.driver.jdbc.core.driver.ShardingSphereURLManager"},
"name":"org.apache.shardingsphere.driver.jdbc.core.driver.spi.classpath.ClasspathURLProvider",
"methods":[{"name":"<init>","parameterTypes":[] }]
},
{
"condition":{"typeReachable":"org.apache.shardingsphere.driver.jdbc.core.driver.ShardingSphereURLManager"},
"name":"org.apache.shardingsphere.driver.jdbc.core.driver.spi.classpath.ClasspathWithEnvironmentURLProvider",
"methods":[{"name":"<init>","parameterTypes":[] }]
},
{
"condition":{"typeReachable":"org.apache.shardingsphere.driver.jdbc.core.driver.ShardingSphereURLManager"},
"name":"org.apache.shardingsphere.driver.jdbc.core.driver.spi.classpath.ClasspathWithSystemPropsURLProvider",
"methods":[{"name":"<init>","parameterTypes":[] }]
},
{
"condition":{"typeReachable":"org.apache.shardingsphere.encrypt.rule.EncryptRule"},
"name":"org.apache.shardingsphere.encrypt.algorithm.assisted.MD5AssistedEncryptAlgorithm",
Expand Down Expand Up @@ -979,7 +954,7 @@
"methods":[{"name":"<init>","parameterTypes":[] }]
},
{
"condition":{"typeReachable":"org.apache.shardingsphere.infra.parser.cache.SQLStatementCacheLoader"},
"condition":{"typeReachable":"org.apache.shardingsphere.sql.parser.core.database.visitor.SQLStatementVisitorFactory"},
"name":"org.apache.shardingsphere.sql.parser.mysql.visitor.statement.type.MySQLDMLStatementVisitor",
"methods":[{"name":"<init>","parameterTypes":[] }]
},
Expand All @@ -1004,7 +979,7 @@
"methods":[{"name":"<init>","parameterTypes":[] }]
},
{
"condition":{"typeReachable":"org.apache.shardingsphere.infra.parser.cache.SQLStatementCacheLoader"},
"condition":{"typeReachable":"org.apache.shardingsphere.sql.parser.core.database.visitor.SQLStatementVisitorFactory"},
"name":"org.apache.shardingsphere.sql.parser.opengauss.visitor.statement.type.OpenGaussDMLStatementVisitor",
"methods":[{"name":"<init>","parameterTypes":[] }]
},
Expand Down Expand Up @@ -1034,7 +1009,7 @@
"methods":[{"name":"<init>","parameterTypes":[] }]
},
{
"condition":{"typeReachable":"org.apache.shardingsphere.infra.parser.cache.SQLStatementCacheLoader"},
"condition":{"typeReachable":"org.apache.shardingsphere.sql.parser.core.database.visitor.SQLStatementVisitorFactory"},
"name":"org.apache.shardingsphere.sql.parser.postgresql.visitor.statement.type.PostgreSQLDMLStatementVisitor",
"methods":[{"name":"<init>","parameterTypes":[] }]
},
Expand Down Expand Up @@ -1124,7 +1099,7 @@
"methods":[{"name":"<init>","parameterTypes":[] }]
},
{
"condition":{"typeReachable":"org.apache.shardingsphere.infra.parser.cache.SQLStatementCacheLoader"},
"condition":{"typeReachable":"org.apache.shardingsphere.sql.parser.core.database.visitor.SQLStatementVisitorFactory"},
"name":"org.apache.shardingsphere.sql.parser.sqlserver.visitor.statement.type.SQLServerDMLStatementVisitor",
"methods":[{"name":"<init>","parameterTypes":[] }]
},
Expand Down Expand Up @@ -1191,6 +1166,16 @@
"condition":{"typeReachable":"org.apache.shardingsphere.infra.yaml.config.shortcut.YamlRuleConfigurationShortcuts"},
"name":"org.apache.shardingsphere.traffic.yaml.config.YamlTrafficRuleConfiguration"
},
{
"condition":{"typeReachable":"org.apache.shardingsphere.transaction.ShardingSphereTransactionManagerEngine"},
"name":"org.apache.shardingsphere.transaction.base.seata.at.SeataATShardingSphereTransactionManager",
"methods":[{"name":"<init>","parameterTypes":[] }]
},
{
"condition":{"typeReachable":"org.apache.shardingsphere.infra.executor.sql.hook.SPISQLExecutionHook"},
"name":"org.apache.shardingsphere.transaction.base.seata.at.SeataTransactionalSQLExecutionHook",
"methods":[{"name":"<init>","parameterTypes":[] }]
},
{
"condition":{"typeReachable":"org.apache.shardingsphere.transaction.ShardingSphereTransactionManagerEngine"},
"name":"org.apache.shardingsphere.transaction.xa.XAShardingSphereTransactionManager",
Expand Down
19 changes: 10 additions & 9 deletions .../src/main/java/org/apache/shardingsphere/infra/url/type/classpath/ClassPathURLLoader.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,11 @@
import lombok.SneakyThrows;
import org.apache.shardingsphere.infra.url.ShardingSphereURLLoader;

import java.io.File;
import java.io.BufferedReader;
import java.io.IOException;
import java.net.URISyntaxException;
import java.nio.file.Files;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.nio.charset.StandardCharsets;
import java.util.Objects;
import java.util.Properties;
import java.util.stream.Collectors;
Expand All @@ -36,12 +37,12 @@ public final class ClassPathURLLoader implements ShardingSphereURLLoader {
@Override
@SneakyThrows(IOException.class)
public String load(final String configurationSubject, final Properties queryProps) {
return Files.readAllLines(getResourceFile(configurationSubject).toPath()).stream().collect(Collectors.joining(System.lineSeparator()));
}

@SneakyThrows(URISyntaxException.class)
private File getResourceFile(final String configurationSubject) {
return new File(Objects.requireNonNull(Thread.currentThread().getContextClassLoader().getResource(configurationSubject)).toURI().getPath());
try (InputStream inputStream = Thread.currentThread().getContextClassLoader().getResourceAsStream(configurationSubject)) {
Objects.requireNonNull(inputStream);
try (BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream, StandardCharsets.UTF_8))) {
return reader.lines().collect(Collectors.joining(System.lineSeparator()));
}
}
}

@Override
Expand Down
18 changes: 6 additions & 12 deletions ...rg/apache/shardingsphere/mode/repository/standalone/jdbc/sql/JDBCRepositorySQLLoader.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -85,15 +85,10 @@ public static JDBCRepositorySQL load(final String type) {
}

/**
* Under the GraalVM Native Image, although there is
* `com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider`, the corresponding
* `com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystem` does not autoload. This is mainly to align the
* behavior of `ZipFileSystemProvider`, so ShardingSphere need to manually open and close the FileSystem
* corresponding to the `resource:/` scheme. For more background reference <a href="https://github.com/oracle/graal/issues/7682">oracle/graal#7682</a>.
* <p/>
* ShardingSphere use the System Property of `org.graalvm.nativeimage.imagecode` to identify whether this class is in the
* GraalVM Native Image environment. The background of this property comes from
* <a href="https://junit.org/junit5/docs/5.10.0/api/org.junit.jupiter.api/org/junit/jupiter/api/condition/DisabledInNativeImage.html">Annotation Interface DisabledInNativeImage</a>.
* Under the GraalVM Native Image, `com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystem` does not autoload.
* This is mainly to align the behavior of `jdk.nio.zipfs.ZipFileSystem`,
* so ShardingSphere need to manually open and close the FileSystem corresponding to the `resource:/` scheme.
* For more background reference <a href="https://github.com/oracle/graal/issues/7682">oracle/graal#7682</a>.
*
* @param url url
* @param type type of JDBC repository SQL
Expand All @@ -104,13 +99,12 @@ public static JDBCRepositorySQL load(final String type) {
* @see sun.nio.fs.UnixFileSystemProvider
*/
private static JDBCRepositorySQL loadFromDirectory(final URL url, final String type) throws URISyntaxException, IOException {
if (null == System.getProperty("org.graalvm.nativeimage.imagecode") || !"runtime".equals(System.getProperty("org.graalvm.nativeimage.imagecode"))) {
return loadFromDirectoryLegacy(url, type);
} else {
if ("resource".equals(url.getProtocol())) {
try (FileSystem ignored = FileSystems.newFileSystem(URI.create("resource:/"), Collections.emptyMap())) {
return loadFromDirectoryInNativeImage(url, type);
}
}
return loadFromDirectoryLegacy(url, type);
}

/**
Expand Down
Loading

0 comments on commit 705cd2d

Please sign in to comment.