Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Real-time replication - switch to riak_kv_overflow_queue #1817

Closed
martinsumner opened this issue Mar 9, 2022 · 0 comments
Closed

Real-time replication - switch to riak_kv_overflow_queue #1817

martinsumner opened this issue Mar 9, 2022 · 0 comments
Assignees

Comments

@martinsumner
Copy link
Contributor

The riak_kv_overflow_queue has been added as part of this enhancement to the reaper/eraser queues.

Currently the realtime replication queue is in memory, and of a limited fix size. Change this to use the overflow queue, to allow for:

  • larger backlogs;
  • backlogs to persist across reboots.

The reason why originally this was simply an in-memory queue, was that full-sync will always fill the gaps. Although full-sync is now cheaper than key-listing, especially when gaps are small, it is still a slow operation.

If there are o(100K) or greater gaps, full-sync is going to be more time consuming and expensive than draining from a queue, if such a queue exists.

In implementing this, the queue should persist across backlogs but start empty. There should be an admin command to empty a previous real-time queue on the node, so that replaying such a persisted backlog is an operator choice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant