sinkv2 (ticdc): limit batch dml size #7960
Conversation
|
[REVIEW NOTIFICATION] This pull request has been approved by:
To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. Reviewer can indicate their review by submitting an approval review. |
ti-chi-bot
added
release-note
|
/run-all-tests |
| event *eventsink.TxnCallbackableEvent, | ||
| tableInfo *timodel.TableInfo, | ||
| spiltUpdate bool, | ||
| ) (insertRows, updateRows, deleteRows []*sqlmodel.RowChange) { | ||
| ) (insertRows, updateRows, deleteRows [][]*sqlmodel.RowChange) { |
There was a problem hiding this comment.
could you add a test case to verify the issue.
There was a problem hiding this comment.
sure, I will do it.
ti-chi-bot
added
the
needs-cherry-pick-release-6.5
…o fix-batch-dml-error
ti-chi-bot
added
size/XL
| convert2RowChanges(row, tableInfo, sqlmodel.RowChangeUpdate)) | ||
| if len(updateRow) >= s.cfg.MaxTxnRow { | ||
| updateRows = append(updateRows, updateRow) | ||
| updateRow = make([]*sqlmodel.RowChange, 0, s.cfg.MaxTxnRow) |
There was a problem hiding this comment.
Why do we use s.cfg.MaxTxnRow here but use preAllocateSize above?
There was a problem hiding this comment.
In the most time len(event) is far more less than s.cfg.MaxTxnRow, it will be a waste if just use s.cfg.MaxTxnRow.
There was a problem hiding this comment.
Thank you for your reminder.
ti-chi-bot
added
status/LGT2
|
/merge |
|
This pull request has been accepted and is ready to merge. Commit hash: a92c641
|
ti-chi-bot
added
the
status/can-merge
|
In response to a cherrypick label: new pull request created to branch |
What problem does this PR solve?
Issue Number: close #7959
What is changed and how it works?
Add a batched dml size limitation. After this PR, the batched dmls in a single SQL will not exceed the
maxTxnRowvalue.Check List
Tests
Questions
Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?
Release note