[Umbrella] CDC DDL Sync Design(Zeta)

[hailin0]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I had searched in the issues and found no similar issues.

Describe the proposal

Backgroud

Currently, we have support for data change capture(CDC #3175), but no further design for schema evolution.

• MySQL
• Oracle
• Postgres
• SQlServer
• ...

And as CDC data synchronization, I think we need to support schema evolution(DDL) as a feature, and I want to hear from you all how you think it can be implemented in SeaTunnel.

Motivation

• Support read database ddl sql (mysql/oracle/...)
• Support ddl sync to any sink（Standardized parse ddl to event）
• Support pause and resume ddl at any time
• Support automatic processing of old data and new data switche

Overall Design

Basic flow

• Depends on checkpoint to push ddl: reader -> writer -> committer

Phase1 - Before Change

• Data flow mixed with structure flow
• DDL is bound to database syntax

Phase2 - Starting Change

• Parsed ddl into schema events

Phase3 - Splitting data flow and structure flow

• Insert checkpoint signals before and after ddl
  • schema-change-before signals
  • schema-change-after signals

Phase4 - Handling schema-change-before signal

• Stop source read data
• Waiting for all old data refresh to sink storage

Phase5 - Execute ddl into source & sink

• Stop checkpoint manager

Phase6 - Handling schema-change-after signal

Phase7 - Completed

• Restart checkpoint manager
• Restart read data for source

Task list

• [Feature][CDC][Zeta] Support schema evolution framework(DDL) #5125
• [Feature][Core] Support cdc task ddl restore for zeta #7463
• [Improve][Connector-V2] Support schema evolution for mysql-cdc and mysql-jdbc #6929
• [Feature][Connector-v2] Support schema evolution for Oracle connector #7908

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[hailin0]

link #3175