[Improve][Connector-V2] Support schema evolution for mysql-cdc and mysql-jdbc #6929

dailai · 2024-05-31T08:35:32Z

Purpose of this pull request

Support schema evolution for mysql-cdc

Support add column
Support drop column
Support modify data type of column
Support rename name of column

Support schema evolution for mysql jdbc sink

Support add column
Support drop column
Support modify data type of column
Support rename name of column

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

...atunnel-connector-v2-e2e/connector-cdc-mysql-e2e/src/test/resources/mysqlcdc_to_console.conf

hailin0 · 2024-05-31T10:05:34Z

subtask of #4672

...est/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/MysqlCDCWithSchemaChangeIT.java

steveGuRen · 2024-06-03T02:15:27Z

中文：
这块是否有方案文档，对这块有兴趣，想了解一下是如何实现点的，想看看能不能实现动态加表的特性，如果可以，想看看我能不能实现
English：
Is there a solution document for this? I am interested in this and would like to understand how it is implemented. I want to see if it is possible to achieve the feature of dynamically adding tables. If so, I would like to see if I can implement it.

...ava/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/config/MySqlSourceConfigFactory.java

.../main/java/org/apache/seatunnel/connectors/cdc/base/schema/AbstractSchemaChangeResolver.java

seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/pom.xml

...apache/seatunnel/connectors/seatunnel/cdc/mysql/source/parser/CustomMySqlAntlrDdlParser.java

.../main/java/org/apache/seatunnel/connectors/cdc/base/schema/AbstractSchemaChangeResolver.java

Carl-Zhou-CN · 2024-06-20T06:43:58Z

Has it already been completed?

seatunnel-api/src/main/java/org/apache/seatunnel/api/table/event/AlterTableColumnEvent.java

hailin0 · 2024-06-25T15:25:52Z

cc @EricJoy2048 @Hisoka-X

Carl-Zhou-CN · 2024-06-26T10:08:05Z

Has it already been completed?

Yes, I have tested both locally and using e2e several times.

ok

...c/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/JdbcDialect.java

.../src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/utils/MySqlTypeUtils.java

Carl-Zhou-CN

LGTM

hailin0

LGTM
cc @EricJoy2048 @Hisoka-X

…sql-jdbc (apache#6929)

…sql-jdbc (apache#6929) try to fix milvus e2e

…sql-jdbc (apache#6929)

Hisoka-X · 2024-09-10T01:55:03Z

Hi @dailai , could you add some doc? I found users even not know this feature so that can not use it at now.

dailai · 2024-09-10T02:03:21Z

OK

hailin0 reviewed May 31, 2024

View reviewed changes

...atunnel-connector-v2-e2e/connector-cdc-mysql-e2e/src/test/resources/mysqlcdc_to_console.conf Outdated Show resolved Hide resolved

hailin0 mentioned this pull request May 31, 2024

[Discuss][CDC] Support processing DDL change events #4672

Closed

3 tasks

hailin0 reviewed May 31, 2024

View reviewed changes

...est/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/MysqlCDCWithSchemaChangeIT.java Outdated Show resolved Hide resolved

...est/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/MysqlCDCWithSchemaChangeIT.java Show resolved Hide resolved

Carl-Zhou-CN reviewed Jun 3, 2024

View reviewed changes

...ava/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/config/MySqlSourceConfigFactory.java Show resolved Hide resolved

hailin0 reviewed Jun 4, 2024

View reviewed changes

.../main/java/org/apache/seatunnel/connectors/cdc/base/schema/AbstractSchemaChangeResolver.java Outdated Show resolved Hide resolved

seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/pom.xml Outdated Show resolved Hide resolved

hailin0 reviewed Jun 4, 2024

View reviewed changes

...apache/seatunnel/connectors/seatunnel/cdc/mysql/source/parser/CustomMySqlAntlrDdlParser.java Show resolved Hide resolved

dailai force-pushed the schema-evolution branch 2 times, most recently from e7930b0 to 920f3d7 Compare June 12, 2024 02:42

dailai changed the title ~~[Improve][Connector-V2] Support schema evolution for mysql-cdc~~ [Improve][Connector-V2] Support schema evolution for mysql-cdc and mysql-jdbc Jun 12, 2024

dailai force-pushed the schema-evolution branch from 920f3d7 to dc49078 Compare June 12, 2024 08:05

hailin0 reviewed Jun 12, 2024

View reviewed changes

.../main/java/org/apache/seatunnel/connectors/cdc/base/schema/AbstractSchemaChangeResolver.java Show resolved Hide resolved

dailai mentioned this pull request Jun 13, 2024

[Hotfix][Zeta] Fix job crash when schema changing on multiple parallel readers #6977

Merged

4 tasks

dailai force-pushed the schema-evolution branch 3 times, most recently from a4b7280 to dc9cbed Compare June 14, 2024 00:36

hailin0 reviewed Jun 15, 2024

View reviewed changes

.../main/java/org/apache/seatunnel/connectors/cdc/base/schema/AbstractSchemaChangeResolver.java Show resolved Hide resolved

dailai mentioned this pull request Jun 17, 2024

[Bug] [CDC] Loss last batch data in splitSchemaChangeStream #7000

Closed

3 tasks

This was referenced Jun 17, 2024

[Hotfix][CDC] Fix split schema change stream #7003

Merged

SeaTunnel community meeting Topic collect #7004

Closed

dailai added 7 commits June 18, 2024 14:52

[Improve][Connector-V2] Support schema evolution for mysql-cdc

36f3dc4

[Improve][Connector-V2] Support schema evolution for jdbc sink

8045086

[Improve][Connector-V2] Fix ci

04ae421

[Improve][Connector-V2] Add test case

235f381

[Improve][Connector-V2] Support table filter

4532a7e

[Improve][Connector-V2] Add test case for modify data type

379c61a

[Improve][Connector-V2] Add test case for table structure vertify

244fe08

dailai force-pushed the schema-evolution branch from b5766d4 to 244fe08 Compare June 18, 2024 06:54

hailin0 added the mysql label Jun 21, 2024

hailin0 reviewed Jun 24, 2024

View reviewed changes

seatunnel-api/src/main/java/org/apache/seatunnel/api/table/event/AlterTableColumnEvent.java Outdated Show resolved Hide resolved

[Improve][Connector-V2] Optimize the code

31f52c5

dailai force-pushed the schema-evolution branch from faf4f27 to 31f52c5 Compare June 24, 2024 10:11

dailai requested a review from hailin0 June 25, 2024 00:52

Carl-Zhou-CN reviewed Jun 26, 2024

View reviewed changes

...c/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/JdbcDialect.java Outdated Show resolved Hide resolved

Carl-Zhou-CN reviewed Jun 26, 2024

View reviewed changes

.../src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/utils/MySqlTypeUtils.java Show resolved Hide resolved

[Improve][Connector-V2] Add comment for the param of method

3cab1ee

dailai force-pushed the schema-evolution branch from 479fb8f to 3cab1ee Compare June 27, 2024 01:42

dailai requested a review from Carl-Zhou-CN June 27, 2024 08:37

Carl-Zhou-CN approved these changes Jun 28, 2024

View reviewed changes

github-actions bot added approved reviewed labels Jun 28, 2024

hailin0 requested review from EricJoy2048 and Hisoka-X July 4, 2024 13:15

hailin0 reviewed Jul 4, 2024

View reviewed changes

EricJoy2048 approved these changes Jul 5, 2024

View reviewed changes

EricJoy2048 merged commit cf91e51 into apache:dev Jul 5, 2024
5 checks passed

hawk9821 pushed a commit to hawk9821/seatunnel that referenced this pull request Jul 10, 2024

[Improve][Connector-V2] Support schema evolution for mysql-cdc and my…

a47ad26

…sql-jdbc (apache#6929)

Thomas-HuWei pushed a commit to zilliztech/seatunnel that referenced this pull request Jul 10, 2024

[Improve][Connector-V2] Support schema evolution for mysql-cdc and my…

32d19be

…sql-jdbc (apache#6929) try to fix milvus e2e

hawk9821 pushed a commit to hawk9821/seatunnel that referenced this pull request Jul 13, 2024

[Improve][Connector-V2] Support schema evolution for mysql-cdc and my…

6b213dc

…sql-jdbc (apache#6929)

hailin0 mentioned this pull request Jul 14, 2024

[Feature][CDC] Support DDL parse for oracle-cdc #7204

Closed

3 tasks

dailai deleted the schema-evolution branch August 1, 2024 00:43

chaorongzhi pushed a commit to chaorongzhi/seatunnel that referenced this pull request Aug 21, 2024

[Improve][Connector-V2] Support schema evolution for mysql-cdc and my…

e38efcb

…sql-jdbc (apache#6929)

dailai mentioned this pull request Sep 10, 2024

[Improve][Doc] Add doc for mysql-cdc schema evolution #7626

Merged

4 tasks

hailin0 mentioned this pull request Oct 28, 2024

[Umbrella] CDC DDL Sync Design(Zeta) #7930

Open

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Improve][Connector-V2] Support schema evolution for mysql-cdc and mysql-jdbc #6929

[Improve][Connector-V2] Support schema evolution for mysql-cdc and mysql-jdbc #6929

dailai commented May 31, 2024 •

edited

Loading

hailin0 commented May 31, 2024

steveGuRen commented Jun 3, 2024

Carl-Zhou-CN commented Jun 20, 2024

hailin0 commented Jun 25, 2024

Carl-Zhou-CN commented Jun 26, 2024 •

edited

Loading

Carl-Zhou-CN left a comment

hailin0 left a comment

Hisoka-X commented Sep 10, 2024

dailai commented Sep 10, 2024

[Improve][Connector-V2] Support schema evolution for mysql-cdc and mysql-jdbc #6929

[Improve][Connector-V2] Support schema evolution for mysql-cdc and mysql-jdbc #6929

Conversation

dailai commented May 31, 2024 • edited Loading

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

hailin0 commented May 31, 2024

steveGuRen commented Jun 3, 2024

Carl-Zhou-CN commented Jun 20, 2024

hailin0 commented Jun 25, 2024

Carl-Zhou-CN commented Jun 26, 2024 • edited Loading

Carl-Zhou-CN left a comment

Choose a reason for hiding this comment

hailin0 left a comment

Choose a reason for hiding this comment

Hisoka-X commented Sep 10, 2024

dailai commented Sep 10, 2024

dailai commented May 31, 2024 •

edited

Loading

Carl-Zhou-CN commented Jun 26, 2024 •

edited

Loading