Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix resuming binlog streaming #156 #160

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

kolbitsch-lastline
Copy link
Contributor

A mysql replication event for changing data is always started by
sending a TableMapEvent (describing the table to be changed),
followed by one or more RowsEvent (containing the data to be
changed). If multiple consecutive RowsEvent events are sent for the
same table, the TableMapEvent is typically skipped (after sending it
once), meaning that resuming from such a binlog is not possible.

Resuming at a binlog position between the TableMapEvent and a
RowsEvent is not possible, as the binlog streamer would miss the
table definition for following DML statements.

This commit tracks the position of the most recently processed
TableMapEvent and uses this for resuming (skipping any actual events
between the last write and resume position that were already
processed).

@kolbitsch-lastline kolbitsch-lastline force-pushed the resume-position-without-table-map-event-156 branch from 9a7999c to fb6b953 Compare March 22, 2020 17:16
A mysql replication event for changing data is always started by
sending a TableMapEvent (describing the table to be changed),
followed by one or more RowsEvent (containing the data to be
changed). If multiple consecutive RowsEvent events are sent for the
same table, the TableMapEvent is typically skipped (after sending it
once), meaning that resuming from such a binlog is not possible.

Resuming at a binlog position between the TableMapEvent and a
RowsEvent is not possible, as the binlog streamer would miss the
table definition for following DML statements.

This commit tracks the position of the most recently processed
TableMapEvent and uses this for resuming (skipping any actual events
between the last write and resume position that were already
processed).

Change-Id: I2bef401ba4f1c0d5f50f177f48c2e866bb411f5d
@fjordan
Copy link
Contributor

fjordan commented Apr 8, 2020

👋 Apologies for the delay in reviews here.

Thank you for your contribution - this does indeed solve the problem you're describing.

After running into this problem while doing some binlog streaming against the target for a separate and new feature we're looking to soon add, I took a slightly different approach that led to the same basic resolution, but with fewer changes (primarily by moving the last resumable position to the DMLEvent instead of creating a new BinlogEvent struct). You can see thee proposed alternative here: #168

I also borrowed your test and slightly modified the assertions a bit, but I wanted to make sure you knew I was intentionally reusing your work 🙏 😄

@kolbitsch-lastline
Copy link
Contributor Author

glad to hear it was useful and thanks for the pointer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants