feat: add an Arrow-based, columnar binlog buffer #91
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
link: vitess.io/vitess/go/hack: invalid reference to runtime.roundupsize
fanyang01
changed the title
fanyang01
changed the title
fanyang01
changed the title
fanyang01
changed the title
GaoYusong
reviewed
Sep 29, 2024
GaoYusong
reviewed
Sep 29, 2024
GaoYusong
reviewed
Sep 29, 2024
GaoYusong
approved these changes
Sep 29, 2024
|
Thanks for your detailed review! The next step would be letting the delta appender buffer as much data as possible until 1) the memory usage becomes too big; 2) some of the tables are queried; 3) a pre-defined time ticker (e.g., every 1 minute) is fired. |
sounds great! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces a columnar delta buffer to store the DML change logs replicated from the primary server. The change logs are transmitted via the ROWS_EVENT binlog events, which include INSERT, UPDATE, and DELETE operations.
This PR is the first step towards resolving issues #55 and #56. Currently, the buffer is flushed to DuckDB for each binlog event, which is not the intended usage. The primary goal of this PR is to ensure that the new components function as expected and pass the existing tests. I will improve the scheduling of buffer flushing in forthcoming PRs.
To enhance the performance of the columnar delta buffer, the following changes have been made:
binlog/rbr.go). This modification minimizes copying and allocation, which is crucial for high performance.replica/controller.go). This is key to avoiding issue Row-by-Row INSERTs are very slow in DuckDB #55. This implementation can be further optimized to a zero-copy approach once PR #283 is merged.dolthub/vitessto the officialvitess.io/vitessfor binlog handling. The official repository has been refactored since the DoltHub fork and thus handles JSON data better and is easier to use. Since our project depends onvitess.io/vitessanddolthub/go-mysql-server, and the latter depends ondolthub/vitess, I have forkedgo-mysql-serveranddolthub/vitessto our organization to resolve the conflicts.This PR has passed all existing binlog replication tests.