Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TiCDC] support data compression when syncing to MySQL/TiDB #9733

Open
zhangjinpeng87 opened this issue Sep 13, 2023 · 3 comments
Open

[TiCDC] support data compression when syncing to MySQL/TiDB #9733

zhangjinpeng87 opened this issue Sep 13, 2023 · 3 comments
Labels
help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. type/feature Issues about a new feature

Comments

@zhangjinpeng87
Copy link
Contributor

Is your feature request related to a problem?

When deploy multiple TiDB Clusters across multiple regions, TiCDC will be used to replicate data changes between these TiDB Clusters. There are some points need to be considered when doing cross region replication:

  • The network bandwidth of cross regions is limited.
  • The cost of data transfer across regions is higher than data transfer inside a region.

Describe the feature you'd like

Support compression when replicate data changes across regions.
MySQL protocol support compression: https://dev.mysql.com/doc/refman/8.0/en/connection-compression-control.html
TiDB also support connection compression: pingcap/tidb#22605

Describe alternatives you've considered

  1. Don't support compression, pay more cost for cross region replication; but it is easier to hit cross region network bandwidth limitation when the data replication throughput is large enough.

Teachability, Documentation, Adoption, Migration Strategy

No response

@zhangjinpeng87 zhangjinpeng87 added the type/feature Issues about a new feature label Sep 13, 2023
@zhangjinpeng87
Copy link
Contributor Author

MySQL go-sql-driver doesn't support compression yet. https://github.com/go-sql-driver/mysql/blob/43e9bef05581335f84d246aba6211af1b5133aae/dsn.go#L468

@zhangjinpeng87 zhangjinpeng87 added the help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. label Sep 13, 2023
@lance6716
Copy link
Contributor

Another less popular MySQL driver has implemented compression go-mysql-org/go-mysql#787 . Hope it's helpful for the contributor who wants to implement it in go-sql-driver

@dveeden
Copy link
Contributor

dveeden commented Oct 19, 2023

MySQL go-sql-driver doesn't support compression yet. https://github.com/go-sql-driver/mysql/blob/43e9bef05581335f84d246aba6211af1b5133aae/dsn.go#L468

Indeed: go-sql-driver/mysql#24

Note that if we implement this with either driver we should aim to support both zlib and zstd.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. type/feature Issues about a new feature
Projects
None yet
Development

No branches or pull requests

3 participants