Phil/recoverylog pruning #347
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Updates the `gazctl shards recover` subcommand to tolerate missing FSM Hints. Previously, this command would fail if it could not find primary hints for the given shard. This was not necessary, since recovery is designed to work without any hints. This changes the `shards recover` subcommand to use the same logic that's used in `recovery.go` for resolving a final set of hints, which may be empty except for the journal name. Now, you can run the command on a shard with only backup hints, or no hints at all, and it will still recover an identical state.
The `shards prune` subcommand was failing with an error if any selected shard was missing backup hints. This updates the subcommand to instead skip pruning any journals where a shard that uses it is missing hints. Put another way, if any selected shard is missing hints for a given recovery log journal, we do not prune that journal.
psFried
added a commit
to estuary/flow
that referenced
this pull request
Aug 23, 2023
This is to pull in a few fixes from gazette/core#347
psFried
added a commit
to estuary/flow
that referenced
this pull request
Aug 23, 2023
This is to pull in a few fixes from gazette/core#347
psFried
added a commit
to estuary/flow
that referenced
this pull request
Aug 23, 2023
This is to pull in a few fixes from gazette/core#347
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes a couple of issues in
gazctlsubcommands regarding missing FSM hints for recovery logs.See individual commit messages.
I tested these both as best I could manually, but there's still no automated tests for them AFAIK. Here's the scenarios I tested, and please feel free to suggest any others you can think of:
Recover:
shards recoveron a few active Flow shards from productionRecover 2:
shards recover --dir ashards recover --dir banddiffto confirm it's the same asa(previously, this would have errored)shards recover --dir canddiffto confirm it's the same asaPrune:
--dry-runcommands against Flow's combustible-cronut environment--dry-run -l estuary.dev/task-type=derivationand got:bytesKept=4606810948793 bytesPruned=26377008674102 bytesTotal=30983819622895 fragmentsKept=17359 fragmentsPruned=102500 fragmentsTotal=119859 shardsTotal=114 skippedJournals=2--dry-run -l estuary.dev/task-type=materializationand got:bytesKept=37440051630 bytesPruned=536165477654 bytesTotal=573605529284 fragmentsKept=3503 fragmentsPruned=9307 fragmentsTotal=12810 shardsTotal=291 skippedJournals=61--dry-run -l estuary.dev/task-type=captureand got:bytesKept=1529386200947 bytesPruned=14368880429210 bytesTotal=15898266630157 fragmentsKept=7896 fragmentsPruned=77030 fragmentsTotal=84926 shardsTotal=512 skippedJournals=56This change is