[Feature][Jdbc-Connector] Add OceanBase Connector #4626

changhuyan · 2023-04-20T02:43:19Z

Purpose of this pull request

Add OceanBase Connector

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to [connector-v2]
  Examples are as follows：
  env {
  job.name = "SeaTunnel"
  spark.executor.instances = 1
  spark.executor.cores = 1
  spark.executor.memory = "1g"
  spark.master = local
  }

source {
jdbc{
url = "jdbc:oceanbase://xxxxx:2881/test"
driver = "com.alipay.oceanbase.jdbc.Driver"
user = "xxx"
password = "xxx"
query = "select id,name from test"
driver_type = "mysql"
}
}

transform {
}

sink {
jdbc {
url = "jdbc:oceanbase://xxx:2881/test"
driver = "com.alipay.oceanbase.jdbc.Driver"
user = "root"
password = "xxxxx"
driver_type = "oracle"
query = "insert into test3(id,name) values(?,?)"
}
}
3. Update plugin-mapping.properties and add new connector information in it
4. Update the pom file of seatunnel-dist

Update the release-note.

hailin0

hailin0 · 2023-04-20T02:49:04Z

...ngine-examples/src/main/java/org/apache/seatunnel/example/engine/SeaTunnelEngineExample.java

@@ -31,7 +31,7 @@ public class SeaTunnelEngineExample {

    public static void main(String[] args)
            throws FileNotFoundException, URISyntaxException, CommandException {
-        String configurePath = args.length > 0 ? args[0] : "/examples/fake_to_console.conf";
+        String configurePath = args.length > 0 ? args[0] : "/examples/oceabase.conf";


hailin0 · 2023-04-20T02:49:08Z

...ctor-v2-example/src/main/java/org/apache/seatunnel/example/spark/v2/SeaTunnelApiExample.java

@@ -26,7 +26,7 @@ public class SeaTunnelApiExample {

    public static void main(String[] args)
            throws FileNotFoundException, URISyntaxException, CommandException {
-        String configurePath = args.length > 0 ? args[0] : "/examples/spark.batch.conf";
+        String configurePath = args.length > 0 ? args[0] : "/examples/oceabase.conf";


hailin0 · 2023-04-20T03:48:15Z

please check ci error
https://github.com/apache/incubator-seatunnel/actions/runs/4749964848/jobs/8438153776?pr=4626

hailin0 · 2023-04-21T06:51:50Z

please check ci error

hailin0 · 2023-04-21T06:53:29Z

...ctor-v2-example/src/main/java/org/apache/seatunnel/example/spark/v2/SeaTunnelApiExample.java

@@ -26,7 +26,7 @@ public class SeaTunnelApiExample {

    public static void main(String[] args)
            throws FileNotFoundException, URISyntaxException, CommandException {
-        String configurePath = args.length > 0 ? args[0] : "/examples/spark.batch.conf";
+        String configurePath = args.length > 0 ? args[0] : "/examples/fake_to_console.conf";


hailin0 · 2023-04-21T07:10:20Z

seatunnel-connectors-v2/connector-jdbc/pom.xml

@@ -144,7 +151,10 @@
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
        </dependency>
-
+        <dependency>


add this drive into：

https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-dist/pom.xml#L558
https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-dist/src/main/assembly/assembly-bin-ci.xml#L195

hailin0 · 2023-04-21T07:10:55Z

...unnel/connectors/seatunnel/jdbc/internal/dialect/oceanbase/OceanBaseMySqlDialectFactory.java

+    public boolean acceptsURL(String url, Optional<String> driverTye) {
+        return url.startsWith("jdbc:oceanbase:")
+                && driverTye.isPresent()
+                && driverTye.get().equalsIgnoreCase("mysql");


add example into docs

1、Modify ci and code style error 2、example into docs 3、add this drive into

hailin0 · 2023-04-22T14:15:02Z

check ci
add e2e testcase

hailin0 · 2023-04-25T02:17:49Z

...e2e-part-2/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcOceanbaseIT.java

+                    + "  `age` varchar(255)  DEFAULT NULL,\n"
+                    + "  `name` varchar(255) DEFAULT NULL\n"


Test all datatypes

hailin0 · 2023-04-25T02:18:46Z

...e2e-part-2/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcOceanbaseIT.java

+    }
+
+    @Override
+    void compareResult() {}


implement this logic

hailin0 · 2023-04-25T02:35:46Z

docs/en/connector-v2/sink/Jdbc.md

@@ -33,6 +33,7 @@ support `Xa transactions`. You can set `is_exactly_once=true` to enable it.
 | user                                      | String  | No       | -             |
 | password                                  | String  | No       | -             |
 | query                                     | String  | No       | -             |
+| driver_type                               | String  | No       | -             |


add appendix & axample

https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/connector-v2/source/Jdbc.md#appendix

hailin0 · 2023-04-27T02:13:55Z

...e2e-part-2/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcOceanbaseIT.java

+import java.util.List;
+import java.util.Map;
+
+@Disabled("Disabled because it needs user's personal oceanbase account to run this test!")


Use docker deploy ob server

https://en.oceanbase.com/docs/community-observer-en-10000000000829647#Use%20Docker%20to%20deploy%20OceanBase%20Database

NickCodeJourney · 2023-05-28T06:20:42Z

why not use mysql connector

changhuyan · 2023-05-29T01:51:21Z

 Because OceanBase databases are used in two ways, namely mysql and oracle.

…

------------------ 原始邮件 ------------------ 发件人: "apache/seatunnel" ***@***.***>; 发送时间: 2023年5月28日(星期天) 下午2:20 ***@***.***>; ***@***.******@***.***>; 主题: Re: [apache/seatunnel] [Feature][Jdbc-Connector] Add OceanBase Connector (PR #4626) why not use mysql connector — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

whhe · 2023-06-27T11:11:40Z

...java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/JdbcDialectFactory.java

@@ -33,7 +35,7 @@ public interface JdbcDialectFactory {
     * @return <code>true</code> if this dialect understands the given URL; <code>false</code>
     *     otherwise.
     */
-    boolean acceptsURL(String url);


I think the driverType param shoud be passed in the create method below, and we can use a new default method to keep the other dialects unchanged.

default JdbcDialect create(String driverType) { return create(); }

whhe · 2023-06-27T11:51:56Z

...unnel/connectors/seatunnel/jdbc/internal/dialect/oceanbase/OceanBaseMySqlDialectFactory.java

+
+    @Override
+    public JdbcDialect create() {
+        return new MysqlDialect();


We can merge the two dialect classes of OceanBase and return the JdbcDialect by the driverType param here.

public JdbcDialect create(@Nonnull String driverType) { if ("mysql".equalsIgnoreCase(driverType)) { return new MysqlDialect(); } return new OracleDialect(); }

whhe · 2023-06-27T11:56:46Z

seatunnel-connectors-v2/connector-jdbc/pom.xml

+                <artifactId>oceanbase-client</artifactId>
+                <version>${oceanbase.version}</version>
+                <scope>provided</scope>
+            </dependency>


It should not be imported directly due to the lgpl license https://github.com/oceanbase/obconnector-j

whhe · 2023-06-27T12:00:37Z

seatunnel-dist/pom.xml

+                    <artifactId>oceanbase-client</artifactId>
+                    <version>${oceanbase.version}</version>
+                    <scope>provided</scope>
+                </dependency>


Should not use the previous internal version, just keep it same with seatunnel-connectors-v2/connector-jdbc/pom.xml

whhe · 2023-06-27T12:03:54Z

...e2e-part-2/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcOceanbaseIT.java

+
+@Disabled("Disabled because it needs user's personal oceanbase account to run this test!")
+public class JdbcOceanbaseIT extends AbstractJdbcIT {
+    private static final String OCEANBASE_IMAGE = "shihd/oceanbase:1.0";


You can use the docker image oceanbase/oceanbase-ce:4.0.0.0 with user root and empty password.

changhuyan · 2023-06-27T12:10:18Z

Yes, there's a better way. You can change it.

…

------------------ 原始邮件 ------------------ 发件人: "apache/seatunnel" ***@***.***>; 发送时间: 2023年6月27日(星期二) 晚上8:04 ***@***.***>; ***@***.******@***.***>; 主题: Re: [apache/seatunnel] [Feature][Jdbc-Connector] Add OceanBase Connector (PR #4626) @whhe requested changes on this pull request. In seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/JdbcDialectFactory.java: > @@ -33,7 +35,7 @@ public interface JdbcDialectFactory { * @return <code>true</code> if this dialect understands the given URL; <code>false</code> * otherwise. */ - boolean acceptsURL(String url); I think the driverType param shoud be passed in the create method below, and we can use a new default method to keep the other dialects unchanged. default JdbcDialect create(String driverType) { return create(); } In seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/oceanbase/OceanBaseMySqlDialectFactory.java: > +import com.google.auto.service.AutoService; + +import java.util.Optional; + ***@***.***(JdbcDialectFactory.class) +public class OceanBaseMySqlDialectFactory implements JdbcDialectFactory { + @OverRide + public boolean acceptsURL(String url, Optional<String> driverTye) { + return url.startsWith("jdbc:oceanbase:") + && driverTye.isPresent() + && driverTye.get().equalsIgnoreCase("mysql"); + } + + @OverRide + public JdbcDialect create() { + return new MysqlDialect(); We can merge the two dialect classes of OceanBase and return the JdbcDialect by the driverType param here. public JdbcDialect ***@***.*** String driverType) { if ("mysql".equalsIgnoreCase(driverType)) { return new MysqlDialect(); } return new OracleDialect(); } In seatunnel-connectors-v2/connector-jdbc/pom.xml: > @@ -129,6 +130,12 @@ <version>${vertica.version}</version> <scope>provided</scope> </dependency> + <dependency> + <groupId>com.oceanbase</groupId> + <artifactId>oceanbase-client</artifactId> + <version>${oceanbase.version}</version> + <scope>provided</scope> + </dependency> It should not be imported directly due to the lgpl license https://github.com/oceanbase/obconnector-j In seatunnel-dist/pom.xml: > @@ -488,6 +489,12 @@ </dependency>  + <dependency> + <groupId>com.alipay.oceanbase</groupId> + <artifactId>oceanbase-client</artifactId> + <version>${oceanbase.version}</version> + <scope>provided</scope> + </dependency> Should not use the previous internal version, just keep it same with seatunnel-connectors-v2/connector-jdbc/pom.xml In seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-2/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcOceanbaseIT.java: > +import org.junit.jupiter.api.Disabled; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.containers.output.Slf4jLogConsumer; +import org.testcontainers.utility.DockerImageName; +import org.testcontainers.utility.DockerLoggerFactory; + +import com.google.common.collect.Lists; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + ***@***.***("Disabled because it needs user's personal oceanbase account to run this test!") +public class JdbcOceanbaseIT extends AbstractJdbcIT { + private static final String OCEANBASE_IMAGE = "shihd/oceanbase:1.0"; You can use the docker image oceanbase/oceanbase-ce:4.0.0.0 with user root and empty password. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

whhe · 2023-06-27T12:23:01Z

@changhuyan Thanks for your contributation! I left some comments, and you can touch me at any time if you have any ideas on them.

For the reference to oceanbase-client, I think it may be an appropriate method to guide users to download it manually through the document, but I'm not sure how this affects the integration tests.

whhe · 2023-06-28T03:06:02Z

Yes, there's a better way. You can change it.
…
------------------ 原始邮件 ------------------ 发件人: "apache/seatunnel" @.>; 发送时间: 2023年6月27日(星期二) 晚上8:04 @.>; @.@.>; 主题: Re: [apache/seatunnel] [Feature][Jdbc-Connector] Add OceanBase Connector (PR #4626) @whhe requested changes on this pull request. In seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/JdbcDialectFactory.java: > @@ -33,7 +35,7 @@ public interface JdbcDialectFactory { * @return <code>true</code> if this dialect understands the given URL; <code>false</code> * otherwise. / - boolean acceptsURL(String url); I think the driverType param shoud be passed in the create method below, and we can use a new default method to keep the other dialects unchanged. default JdbcDialect create(String driverType) { return create(); } In seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/oceanbase/OceanBaseMySqlDialectFactory.java: > +import com.google.auto.service.AutoService; + +import java.util.Optional; + @.(JdbcDialectFactory.class) +public class OceanBaseMySqlDialectFactory implements JdbcDialectFactory { + @OverRide + public boolean acceptsURL(String url, Optional<String> driverTye) { + return url.startsWith("jdbc:oceanbase:") + && driverTye.isPresent() + && driverTye.get().equalsIgnoreCase("mysql"); + } + + @OverRide + public JdbcDialect create() { + return new MysqlDialect(); We can merge the two dialect classes of OceanBase and return the JdbcDialect by the driverType param here. public JdbcDialect @.* String driverType) { if ("mysql".equalsIgnoreCase(driverType)) { return new MysqlDialect(); } return new OracleDialect(); } In seatunnel-connectors-v2/connector-jdbc/pom.xml: > @@ -129,6 +130,12 @@ <version>${vertica.version}</version> <scope>provided</scope> </dependency> + <dependency> + <groupId>com.oceanbase</groupId> + <artifactId>oceanbase-client</artifactId> + <version>${oceanbase.version}</version> + <scope>provided</scope> + </dependency> It should not be imported directly due to the lgpl license https://github.com/oceanbase/obconnector-j In seatunnel-dist/pom.xml: > @@ -488,6 +489,12 @@ </dependency>  + <dependency> + <groupId>com.alipay.oceanbase</groupId> + <artifactId>oceanbase-client</artifactId> + <version>${oceanbase.version}</version> + <scope>provided</scope> + </dependency> Should not use the previous internal version, just keep it same with seatunnel-connectors-v2/connector-jdbc/pom.xml In seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-2/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcOceanbaseIT.java: > +import org.junit.jupiter.api.Disabled; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.containers.output.Slf4jLogConsumer; +import org.testcontainers.utility.DockerImageName; +import org.testcontainers.utility.DockerLoggerFactory; + +import com.google.common.collect.Lists; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + @.("Disabled because it needs user's personal oceanbase account to run this test!") +public class JdbcOceanbaseIT extends AbstractJdbcIT { + private static final String OCEANBASE_IMAGE = "shihd/oceanbase:1.0"; You can use the docker image oceanbase/oceanbase-ce:4.0.0.0 with user root and empty password. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.>

Do you have time to complete this work recently? If not, we can also find other developers in the community to continue development based on your work.

[Feature][Jdbc-Connector] Add OceanBase Connector

7e881a3

hailin0 reviewed Apr 20, 2023

View reviewed changes

changhuyan added 2 commits April 20, 2023 20:15

[Feature][Jdbc-Connector]Add Oceanbase Connector

6942728

Merge branch 'dev' into OceanBase-Connector

fc2b53a

hailin0 reviewed Apr 21, 2023

View reviewed changes

changhuyan added 2 commits April 21, 2023 17:13

[Feature][Jdbc-Connector]

4feacd7

1、Modify ci and code style error 2、example into docs 3、add this drive into

[Feature][Jdbc-Connector] Add java.util.*

c2b59a7

changhuyan added 3 commits April 23, 2023 10:01

changhuyan [Feature][Jdbc-Connector] Modify ci and code style error

f007a2b

[Feature][Jdbc-Connector] Add e2e testcase

e9c4289

[Feature][Jdbc-Connector] Add e2etest License

cd2171e

hailin0 reviewed Apr 25, 2023

View reviewed changes

hailin0 reviewed Apr 27, 2023

View reviewed changes

whhe suggested changes Jun 27, 2023

View reviewed changes

whhe mentioned this pull request Jun 28, 2023

[Feature][Connector-V2][Jdbc] Add oceanbase dialect factory #4989

Merged

5 tasks

EricJoy2048 closed this Jul 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Jdbc-Connector] Add OceanBase Connector #4626

[Feature][Jdbc-Connector] Add OceanBase Connector #4626

changhuyan commented Apr 20, 2023 •

edited

Loading

hailin0 left a comment •

edited

Loading

hailin0 Apr 20, 2023

hailin0 Apr 20, 2023

hailin0 commented Apr 20, 2023

hailin0 commented Apr 21, 2023

hailin0 Apr 21, 2023

changhuyan Apr 21, 2023

hailin0 Apr 21, 2023

changhuyan Apr 21, 2023

hailin0 Apr 21, 2023

hailin0 commented Apr 22, 2023

hailin0 Apr 25, 2023

hailin0 Apr 25, 2023

hailin0 Apr 25, 2023

hailin0 Apr 27, 2023

NickCodeJourney commented May 28, 2023

changhuyan commented May 29, 2023 via email

whhe Jun 27, 2023

whhe Jun 27, 2023

whhe Jun 27, 2023

whhe Jun 27, 2023

whhe Jun 27, 2023

changhuyan commented Jun 27, 2023 via email

whhe commented Jun 27, 2023

whhe commented Jun 28, 2023

		+ " `age` varchar(255) DEFAULT NULL,\n"
		+ " `name` varchar(255) DEFAULT NULL\n"

[Feature][Jdbc-Connector] Add OceanBase Connector #4626

[Feature][Jdbc-Connector] Add OceanBase Connector #4626

Conversation

changhuyan commented Apr 20, 2023 • edited Loading

Purpose of this pull request

Check list

hailin0 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hailin0 commented Apr 20, 2023

hailin0 commented Apr 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hailin0 commented Apr 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NickCodeJourney commented May 28, 2023

changhuyan commented May 29, 2023 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

changhuyan commented Jun 27, 2023 via email

whhe commented Jun 27, 2023

whhe commented Jun 28, 2023

changhuyan commented Apr 20, 2023 •

edited

Loading

hailin0 left a comment •

edited

Loading