Replies: 2 comments
-
Using the source table's data size to set the number of buckets for the sink table has two issues:
Supporting independent table property settings for the sink table might be a better approach. |
Beta Was this translation helpful? Give feedback.
0 replies
-
Solved by transform table-optis. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
When I sync multiple MySQL tables to Paimon, the tables have different sizes. Some sinks support using buckets to distribute data, and different tables need different bucket settings. We can use the source data's metadata to automatically set the number of buckets for the sink tables based on their sizes when create sink table.
Could we support table metadata in the
CreateTableEvent
? like TABLE_ROWS, AVG_ROW_LENGTH, DATA_LENGTH, and MAX_DATA_LENGTH.This would allow the sink to automatically set the number of buckets based on the table size when creating the table using the
CreateTableEvent
.Beta Was this translation helpful? Give feedback.
All reactions