[Feature][Kudu] Refactor Kudu functionality and Sink support CDC data. #5437

Carl-Zhou-CN · 2023-09-06T09:46:28Z

1.Refactored the process of Kudu tablet splitting.
2.Adjusted the mapping of error types.
3.Sink Supported CDC Data (Change Data Capture).
4.Supported Kerberos.
5.add e2e

Purpose of this pull request

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

EricJoy2048 · 2023-09-26T12:35:11Z

docs/en/connector-v2/sink/Kudu.md


 ## Key features

+- [x] [batch](../../concept/connector-v2-features.md)


It seems that you have added the features of Source to Sink's documentation

EricJoy2048 · 2023-09-26T12:36:11Z

docs/en/connector-v2/sink/Kudu.md

+| TIMESTAMP           | UNIXTIME_MICROS          |
+| BYTES               | BINARY                   |
+
+## Sink Options


Format have some error:

EricJoy2048 · 2023-09-26T12:37:51Z

docs/en/connector-v2/source/Kudu.md

+| UNIXTIME_MICROS          | TIMESTAMP           |
+| BINARY                   | BYTES               |
+
+## Source Options


Format have some error.

EricJoy2048 · 2023-09-26T12:40:09Z

docs/en/connector-v2/sink/Kudu.md

+    kudu_masters = "kudu-master-cdc:7051"
+    table_name = "kudu_sink_table"
+ }
+}
 ```


Please add examples for kerberos enable.

EricJoy2048 · 2023-09-26T12:45:36Z

docs/en/connector-v2/source/Kudu.md

+    source_table_name = "kudu"
+    kudu_masters = "kudu-master:7051"
+    table_name = "kudu_sink_table"
+ }
 ```


Please add kerberos enable examples.

EricJoy2048 · 2023-10-20T05:55:38Z

...in/java/org/apache/seatunnel/connectors/seatunnel/kudu/source/KuduSourceSplitEnumerator.java

-            parameters[i] = new Long[] {start, end};
-            start = end + 1;
+    private void addPendingSplit(Collection<KuduSourceSplit> splits) {
+        int readerCount = enumeratorContext.currentParallelism();


Need synchronized (stateLock) too.

EricJoy2048 · 2023-10-20T05:57:59Z

...in/java/org/apache/seatunnel/connectors/seatunnel/kudu/source/KuduSourceSplitEnumerator.java

+    @Override
+    public void registerReader(int subtaskId) {
+        log.debug("Register reader {} to KuduSourceSplitEnumerator.", subtaskId);
+        if (!pendingSplits.isEmpty()) {


Need synchronized (stateLock) too.

EricJoy2048 · 2023-10-20T05:59:03Z

...in/java/org/apache/seatunnel/connectors/seatunnel/kudu/source/KuduSourceSplitEnumerator.java

    }

    @Override
    public KuduSourceState snapshotState(long checkpointId) throws Exception {
-        return null;
+        synchronized (stateLock) {
+            return new KuduSourceState(pendingSplits);


Suggested change

return new KuduSourceState(pendingSplits);

return new KuduSourceState(new HashMap<>(pendingSplits));

EricJoy2048 · 2023-10-20T06:23:47Z

...nnector-kudu/src/main/java/org/apache/seatunnel/connectors/seatunnel/kudu/util/KuduUtil.java

+        }
+        Configuration conf = new Configuration();
+        conf.set(HADOOP_AUTH_KEY, KRB);
+        UserGroupInformation.setConfiguration(conf);


UserGroupInformation is s static Class, If there are more than one place use UserGroupInformation.setConfiguration(conf); in one JVM, this will generate mutual coverage. So new a UserGroupInformation Object is more suggested approach.

EricJoy2048 · 2023-10-20T06:34:33Z

docs/en/connector-v2/source/Kudu.md


 ## Key features

 - [x] [batch](../../concept/connector-v2-features.md)
- [ ] [stream](../../concept/connector-v2-features.md)
+- [x] [stream](../../concept/connector-v2-features.md)


From the code it seems not supported stream

EricJoy2048 · 2023-10-21T02:13:47Z

...nnector-kudu/src/main/java/org/apache/seatunnel/connectors/seatunnel/kudu/util/KuduUtil.java

+    }
+
+    private static UserGroupInformation loginAndReturnUgi(CommonConfig config) throws IOException {
+        if (StringUtils.isBlank(config.getPrincipal()) || StringUtils.isBlank(config.getKeytab())) {


synchronized(UserGroupInformation.class) { }

Can fix UGI coverage problem.

hailin0 · 2023-10-25T04:15:00Z

.../connector-kudu-e2e/src/test/java/org/apache/seatunnel/e2e/connector/kudu/KuduCDCSinkIT.java

+        type = {EngineType.FLINK, EngineType.SPARK},
+        disabledReason = "Currently SPARK and FLINK do not support cdc")


remove flink ？

Suggested change

type = {EngineType.FLINK, EngineType.SPARK},

disabledReason = "Currently SPARK and FLINK do not support cdc")

type = {EngineType.SPARK},

disabledReason = "Currently SPARK do not support cdc")

hailin0 · 2023-10-25T04:15:32Z

.../connector-kudu-e2e/src/test/java/org/apache/seatunnel/e2e/connector/kudu/KuduCDCSinkIT.java

+                .untilAsserted(
+                        () -> {
+                            System.out.println(readData(KUDU_SINK_TABLE).size());
+                            Assertions.assertEquals(readData(KUDU_SINK_TABLE).size(), 2);


assert rows & field data？

hailin0 · 2023-10-25T04:16:25Z

LGTM

hailin0

LGTM

EricJoy2048

LGTM if CI complete.

zhouyao and others added 6 commits September 6, 2023 17:38

[Feature][Kudu] Refactor Kudu functionality and support CDC.

359df3d

[Feature][kudu] e2e

d2b819c

[Feature][kudu] e2e

10704b1

Merge branch 'apache:dev' into kudu-re

56dd781

Merge branch 'apache:dev' into kudu-re

c704e39

[Feature][Kudu] doc

083bc46

EricJoy2048 mentioned this pull request Sep 26, 2023

[DOC][Connector-V2][Kudu] update kudu doc #5231

Closed

5 tasks

Carl-Zhou-CN changed the title ~~[Feature][Kudu] Refactor Kudu functionality and support CDC.~~ [Feature][Kudu] Refactor Kudu functionality and Sink support CDC data. Oct 19, 2023

EricJoy2048 reviewed Oct 20, 2023

View reviewed changes

EricJoy2048 reviewed Oct 21, 2023

View reviewed changes

Carl-Zhou-CN and others added 4 commits October 21, 2023 11:54

Merge branch 'apache:dev' into kudu-re

58f7194

fix

6e97cf3

fix

b6d6251

fix

58822c4

hailin0 reviewed Oct 25, 2023

View reviewed changes

zhouyao and others added 2 commits October 25, 2023 15:17

fix

0e3f156

fix

99cba77

hailin0 previously approved these changes Oct 25, 2023

View reviewed changes

github-actions bot added approved reviewed labels Oct 25, 2023

fix

4f88c87

Carl-Zhou-CN dismissed hailin0’s stale review via 4f88c87 October 25, 2023 12:20

github-actions bot removed approved reviewed labels Oct 25, 2023

EricJoy2048 previously approved these changes Oct 25, 2023

View reviewed changes

EricJoy2048 added this to the 2.3.4 milestone Oct 25, 2023

github-actions bot added approved reviewed labels Oct 25, 2023

fix

afd2376

Carl-Zhou-CN dismissed EricJoy2048’s stale review via afd2376 October 25, 2023 14:39

github-actions bot removed approved reviewed labels Oct 25, 2023

hailin0 approved these changes Oct 25, 2023

View reviewed changes

github-actions bot added approved reviewed labels Oct 25, 2023

EricJoy2048 approved these changes Oct 26, 2023

View reviewed changes

EricJoy2048 merged commit 22110eb into apache:dev Oct 26, 2023
6 of 7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Kudu] Refactor Kudu functionality and Sink support CDC data. #5437

[Feature][Kudu] Refactor Kudu functionality and Sink support CDC data. #5437

Carl-Zhou-CN commented Sep 6, 2023 •

edited

Loading

EricJoy2048 Sep 26, 2023

EricJoy2048 Sep 26, 2023

EricJoy2048 Sep 26, 2023

EricJoy2048 Sep 26, 2023

EricJoy2048 Sep 26, 2023

EricJoy2048 Oct 20, 2023

EricJoy2048 Oct 20, 2023

EricJoy2048 Oct 20, 2023

EricJoy2048 Oct 20, 2023

EricJoy2048 Oct 20, 2023

EricJoy2048 Oct 21, 2023

hailin0 Oct 25, 2023

hailin0 Oct 25, 2023

hailin0 commented Oct 25, 2023

hailin0 left a comment

EricJoy2048 left a comment


		## Key features

		- [x] [batch](../../concept/connector-v2-features.md)

	return new KuduSourceState(pendingSplits);
	return new KuduSourceState(new HashMap<>(pendingSplits));

		type = {EngineType.FLINK, EngineType.SPARK},
		disabledReason = "Currently SPARK and FLINK do not support cdc")

[Feature][Kudu] Refactor Kudu functionality and Sink support CDC data. #5437

[Feature][Kudu] Refactor Kudu functionality and Sink support CDC data. #5437

Conversation

Carl-Zhou-CN commented Sep 6, 2023 • edited Loading

Purpose of this pull request

Check list

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hailin0 commented Oct 25, 2023

hailin0 left a comment

Choose a reason for hiding this comment

EricJoy2048 left a comment

Choose a reason for hiding this comment

Carl-Zhou-CN commented Sep 6, 2023 •

edited

Loading