-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
VStream: Skip vindex keyrange filtering when we can #14384
Merged
mattlord
merged 7 commits into
vitessio:main
from
planetscale:vstream_keyrange_optimization
Oct 30, 2023
Merged
VStream: Skip vindex keyrange filtering when we can #14384
mattlord
merged 7 commits into
vitessio:main
from
planetscale:vstream_keyrange_optimization
Oct 30, 2023
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Matt Lord <mattalord@gmail.com>
Review ChecklistHello reviewers! 👋 Please follow this checklist when reviewing this Pull Request. General
Tests
Documentation
New flags
If a workflow is added or modified:
Backward compatibility
|
vitess-bot
bot
added
NeedsDescriptionUpdate
The description is not clear or comprehensive enough, and needs work
NeedsIssue
A linked issue is missing for this Pull Request
NeedsWebsiteDocsUpdate
What it says
labels
Oct 28, 2023
mattlord
added
Component: VReplication
Type: Enhancement
Logical improvement (somewhere between a bug and feature)
labels
Oct 28, 2023
mattlord
force-pushed
the
vstream_keyrange_optimization
branch
6 times, most recently
from
October 29, 2023 00:55
2c45ace
to
4077675
Compare
Signed-off-by: Matt Lord <mattalord@gmail.com>
mattlord
force-pushed
the
vstream_keyrange_optimization
branch
5 times, most recently
from
October 29, 2023 18:37
c959721
to
6effef3
Compare
mattlord
removed
NeedsDescriptionUpdate
The description is not clear or comprehensive enough, and needs work
NeedsWebsiteDocsUpdate
What it says
NeedsIssue
A linked issue is missing for this Pull Request
labels
Oct 29, 2023
mattlord
requested review from
deepthi,
rohit-nayak-ps and
ajm188
as code owners
October 29, 2023 18:55
Signed-off-by: Matt Lord <mattalord@gmail.com>
mattlord
force-pushed
the
vstream_keyrange_optimization
branch
from
October 29, 2023 19:03
6effef3
to
8d19288
Compare
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
dbussink
approved these changes
Oct 30, 2023
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a very useful optimization and we'll also look at optimizing this when we do need to call this as well.
rohit-nayak-ps
approved these changes
Oct 30, 2023
Signed-off-by: Matt Lord <mattalord@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
We were always adding
in_keyrange
filters for a vstream when the target keyspace was sharded. This PR adds a check to see if the shard mappings are 1:1 and the key ranges are equal for both shards — if so then we do not add thein_keyrange
filter.When that filter is in place, every row/binlog event on the VReplication stream source must have its KeyspaceID calculated before adding it to the vstream for the target. This same filter from the VReplication record is also used when a
VDiff
is performed for the workflow. In the case ofVDiff
, this means that each source tablet needs to calculate the KeyspaceID for every record read from its mysqld before adding it to the vstream used for the diff.This additional vindex work can be costly — in particular during the VDiff where you are reading every row from each table that you're diffing in quick succession — and especially so for more complex vindex types such as
unicode_loose_md5
, so skipping that work whenever possible is a good optimization.That is what this PR does.
Related Issue(s)
Checklist