Skip to content
This repository has been archived by the owner on Jun 23, 2022. It is now read-only.

feat(split): child replica apply private logs, in-memory mutations and catch up parent #319

Merged
merged 21 commits into from
Feb 18, 2020

Conversation

hycdong
Copy link
Contributor

@hycdong hycdong commented Sep 23, 2019

Simple partition split process

  1. meta receives client partition split request, and change partition count split: meta start partition split #286
  2. replica notices partition count changed during on_config_sync
  3. parent partition create child partition split: parent replica create child replica #291
  4. parent prepare states for child to learn feat(split): parent replica prepare states #299
  5. child partition async learn states from parent
  6. all child partitions in group finish learn process, parent will stop read and write
  7. meta server register child partitions
  8. child partition active, and parent recover read and write

More partition split discussion in issue #69 and partition split design doc
This pr solves the part of fifth step of partition split, which is bold in process description.

What this pr solved

  • child_apply_private_logs
    • child apply private logs and in-memory mutations
    • similar to function apply_learned_state_from_private_log during learn
  • child_catch_up_states
    • similar to function init_learn with learner_status LearningWithPrepare
    • child will also prepare mutations from parent (which will show in further pr) during async-learn, as a result, child prepare_list last_committed_decree should be equal to parent prepare list last_committed_decree, which is the goal_decree, and local_decree is child local last_committed_decree which is the last decree in async-learn.
    • if local_decree < goal_decree, it means there are mutations written to parent during async-learn, child does not catch up parent, there are still some mutations child not learn.
    • if local_decree >= _prepare_list->min_decree(), it means all missing mutations are all in prepare list, otherwise, some missing mutations have already in private log, should call catch_up_with_private_logs to catch up all missing mutations
  • add is_caught_up flag for partition_split context to identify if child catch up parent

Tests

  • add replay log failed case for child_learn_states
  • add unit tests for child_apply_private_logs and child_catch_up

src/dist/replication/lib/replica_split.cpp Outdated Show resolved Hide resolved
src/dist/replication/lib/replica_split.cpp Outdated Show resolved Hide resolved
src/dist/replication/lib/replica.h Show resolved Hide resolved
src/dist/replication/lib/replica_split.cpp Outdated Show resolved Hide resolved
src/dist/replication/lib/replica_split.cpp Outdated Show resolved Hide resolved
src/dist/replication/lib/replica_split.cpp Outdated Show resolved Hide resolved
src/dist/replication/lib/replica_split.cpp Outdated Show resolved Hide resolved
LPC_CATCHUP_WITH_PRIVATE_LOGS,
tracker(),
[this]() {
catch_up_with_private_logs(partition_status::PS_PARTITION_SPLIT);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

catch_up_with_private_logs 后续 PR 建议改个名字,这个函数和 child_apply_private_logs 对代码不熟悉的很容易看混,而且不明其意

neverchanje
neverchanje previously approved these changes Feb 14, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants