checkpoint: Use core changefeed for fast notifications #1020
Conversation
bobvawter
requested review from
BramGruneir,
sravotto,
Jeremyyang920 and
ZhouXing19
bobvawter
force-pushed
the
bob_checkpoint_feed
branch
from
109d69a
to
b88be33
Compare
There was a problem hiding this comment.
I'm worried this is going to need more testing.
Lots of updates quickly. Failure of the stream for some reason, can we redial?
An option to turn this off.
What do you think?
Reviewed 4 of 4 files at r1, all commit messages.
Reviewable status:complete! all files reviewed, all discussions resolved (waiting on @Jeremyyang920, @sravotto, and @ZhouXing19)
bobvawter
force-pushed
the
bob_checkpoint_feed
branch
from
b88be33
to
c144d69
Compare
This change creates a core changefeed over the checkpoints table to allow multiple Replicator instances to notify one another to refresh their view of the table. This behavior can be disabled with the `--disableCheckpointStream` flag. The test framework ensures that the staging cluster has rangefeeds enabled to support the split-mode test cases (i.e. where the staging cluster is distinct from the source cluster). H/T: @BramGruneir for the reminder of https://github.com/bobvawter/cacheroach/blob/512fae9092fc1e48406ef38fbb9959fc749d607f/pkg/store/cdc/cdc.go
bobvawter
force-pushed
the
bob_checkpoint_feed
branch
from
c144d69
to
0086da6
Compare
There was a problem hiding this comment.
PTAL. Added a CLI flag and plumbing to disable the feed. Added a test to kill the feed's DB connection and verify that it restarts.
Rapid updates wind up getting debounced in the fastWakeup loop; the call to notify will close the previously-observed channel and it won't be observed again until the refresh has completed. This isn't really any different from an instance receiving multiple resolved-timestamp messages back-to-back.
Reviewable status: 0 of 8 files reviewed, all discussions resolved (waiting on @BramGruneir, @Jeremyyang920, @sravotto, and @ZhouXing19)
There was a problem hiding this comment.
Reviewed 8 of 8 files at r2, all commit messages.
Reviewable status:
This change creates a core changefeed over the checkpoints table to allow multiple Replicator instances to notify one another to refresh their view of the table.
H/T: @BramGruneir for the reminder of
https://github.com/bobvawter/cacheroach/blob/512fae9092fc1e48406ef38fbb9959fc749d607f/pkg/store/cdc/cdc.go
This change is