feat: filter out trimmed data on compaction #379
Conversation
…am metadata Signed-off-by: Shichao Nie <niesc@automq.com>
Signed-off-by: Shichao Nie <niesc@automq.com>
Qodana for JVMIt seems all right 👌 No new problems were found according to the checks applied 💡 Qodana analysis was run in the pull request mode: only the changed files were checked Contact Qodana teamContact us at qodana-support@jetbrains.com
|
|
The author of this PR, SCNieh, is not an activated member of this organization on Codecov. |
| CompletableFuture<List<StreamMetadata>> streamMetadataFuture = this.streamManager.getOpeningStreams(); | ||
| CompletableFuture<List<S3ObjectMetadata>> objectMetadataFuture = this.objectManager.getServerObjects(); | ||
| return CompletableFuture.allOf(streamMetadataFuture, objectMetadataFuture).thenApplyAsync(v -> { | ||
| List<StreamMetadata> streamMetadataList = streamMetadataFuture.join(); |
There was a problem hiding this comment.
- 我们风格对齐一下，我看大家都习惯把Future命名为xxxCf，也挺好的。
- allOf + join 这个写法总感觉让人不放心，因为join涉及到异常的，是不是aCf.thenCombine(bCf, Pair::new).thenApplyAsync() 更好点。
| @@ -384,6 +428,10 @@ boolean sanityCheckCompactionResult(List<S3ObjectMetadata> objectsToSplit, List< | |||
| for (long objectId : request.getCompactedObjectIds()) { | |||
| S3ObjectMetadata metadata = objectMetadataMap.get(objectId); | |||
| for (StreamOffsetRange streamOffsetRange : metadata.getOffsetRanges()) { | |||
| if (!streamMetadataMap.containsKey(streamOffsetRange.getStreamId()) || | |||
Signed-off-by: Shichao Nie <niesc@automq.com>
| } | ||
| if (streamDataBlock.getStartOffset() < streamMetadataMap.get(streamDataBlock.getStreamId()).getStartOffset()) { |
There was a problem hiding this comment.
Use block.endOffset <= stream.startOffset to filter
| } | ||
| }, executorService)); | ||
| this.scheduledExecutorService.execute(() -> { | ||
| CompletableFuture<List<StreamMetadata>> streamMetadataCf = this.streamManager.getOpeningStreams(); |
There was a problem hiding this comment.
StreamManager#getOpeningStreams will get the current node opening streams
Signed-off-by: Shichao Nie <niesc@automq.com>
|
Merge after this issue closed: #419 |
Signed-off-by: Shichao Nie <niesc@automq.com>
Signed-off-by: Shichao Nie <niesc@automq.com>
Signed-off-by: Shichao Nie <niesc@automq.com>
Signed-off-by: Shichao Nie <niesc@automq.com>
use stream metadata to identify trimmed data blocks instead of object metadata