storage: support MVCC range tombstones in MVCCExportToSST
#82873
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This change is |
erikgrinaker
force-pushed
the
mvcc-range-tombstones-export
branch
2 times, most recently
from
73e464f
to
364ef7d
Compare
|
This should be ready for review now. Sorry about having to drag in #82799. |
itsbilal
approved these changes
Jun 20, 2022
There was a problem hiding this comment.
Reviewed 5 of 6 files at r1, 2 of 8 files at r2, 3 of 6 files at r3, 1 of 1 files at r4, 10 of 11 files at r5, 4 of 4 files at r6, all commit messages.
Reviewable status:complete! 1 of 0 LGTMs obtained (waiting on @aliher1911, @dt, and @erikgrinaker)
pkg/storage/mvcc.go line 4280 at r6 (raw file):
// doesn't end up being any further point keys covered by it and we go on to // flush them as-is at their normal end key. We need to make sure we have // enough MaxSize budget to flush then in all of these cases.
nit: them* ?
pkg/storage/testdata/mvcc_histories/export line 3 at r6 (raw file):
# Tests MVCC export. # # Sets up the following dataset, where x is tombstone, o-o is range tombstone, [] is intent.
nit: MVCC range tombstone* just to make it clear this isn't a Pebble range tombstone.
This patch adds support for exporting MVCC range tombstones in `MVCCExportToSST()`, and by extension in the KV `Export` method. This will only happen after two version gates are enabled: * `EnablePebbleFormatVersionRangeKeys`: begins emitting SSTs in `Pebblev2` format, which supports Pebble range keys. * `MVCCRangeTombstones`: allows writing MVCC range tombstones via KV. MVCC range tombstones are emitted in the same way as point tombstones: all tombstones if `ExportAllRevisions` is enabled, or the latest visible tombstone if `StartTS` is given. MVCC range tombstones are truncated to the SST bounds. For example, if exporting the span `a-f` then any range tombstones wider than the span will be truncated to `[a-f)`. If the export hits a limit e.g. at `c` then any MVCC range tombstones in the returned SST are truncated to `[a-c)`. If `StopMidKey` is enabled, then it's possible for two subsequent exports to contain overlapping MVCC range tombstones. For example, given the range tombstone `[a-f)@5`, if we return a resume key at `c@3` then the response will contain a truncated MVCC range tombstone `[a-c\0)@5` which covers the point keys at `c`, but resuming from `c@3` will contain the MVCC range tombstone `[c-f)@5` which overlaps with the MVCC range tombstone in the previous response for the interval `[c-c\0)@5`. `AddSSTable` will allow this overlap during ingestion once it supports MVCC range tombstones. Release note: None
erikgrinaker
force-pushed
the
mvcc-range-tombstones-export
branch
from
364ef7d
to
a1ecb11
Compare
|
This comes with a moderate performance penalty, which is mostly due to enabling range keys in Pebble and TFTR! bors r=itsbilal |
|
bors r=itsbilal |
|
Build succeeded: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch adds support for exporting MVCC range tombstones in
MVCCExportToSST(), and by extension in the KVExportmethod.This will only happen after two version gates are enabled:
EnablePebbleFormatVersionRangeKeys: begins emitting SSTs inPebblev2format, which supports Pebble range keys.MVCCRangeTombstones: allows writing MVCC range tombstones via KV.MVCC range tombstones are emitted in the same way as point tombstones:
all tombstones if
ExportAllRevisionsis enabled, or the latest visibletombstone if
StartTSis given.MVCC range tombstones are truncated to the SST bounds. For example, if
exporting the span
a-fthen any range tombstones wider than the spanwill be truncated to
[a-f). If the export hits a limit e.g. atcthen any MVCC range tombstones in the returned SST are truncated to
[a-c).If
StopMidKeyis enabled, then it's possible for two subsequentexports to contain overlapping MVCC range tombstones. For example, given
the range tombstone
[a-f)@5, if we return a resume key atc@3thenthe response will contain a truncated MVCC range tombstone
[a-c\0)@5which covers the point keys at
c, but resuming fromc@3will containthe MVCC range tombstone
[c-f)@5which overlaps with the MVCC rangetombstone in the previous response for the interval
[c-c\0)@5.AddSSTablewill allow this overlap during ingestion once it supportsMVCC range tombstones.
Resolves #71398.
Release note: None