-
Notifications
You must be signed in to change notification settings - Fork 308
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor: make region manifest checkpoint ran in background #4133
Conversation
…ncremental files in object store
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4133 +/- ##
==========================================
- Coverage 85.39% 84.67% -0.72%
==========================================
Files 994 1019 +25
Lines 173994 178741 +4747
==========================================
+ Hits 148580 151351 +2771
- Misses 25414 27390 +1976 |
Co-authored-by: Yingwen <realevenyag@gmail.com>
@evenyag @WenyXu @killme2008 PTAL |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. It will help a lot when writing.
@evenyag PTAL |
I hereby agree to the terms of the GreptimeDB CLA.
Refer to a related PR or issue link (optional)
What's changed and what's your intention?
Region manifest checkpoint currently requires many object store reads and writes. The checkpoint is performed synchronously in the region manifest update. When the object store is slow, it could easily cause the region manifest update method executes too long. This PR makes the checkpoint ran in the background (performed asynchronously) to solve the problem.
This PR also makes some changes to how the checkpoint is done. Originally the current checkpoint is the last checkpoint merged with all the incremental files. This process has to scan the object store. However, since the checkpoint is actually a snapshot of the current region manifest, which is already present in the memory, stick to the old ways seems unnecessary. So this PR directly use the region manifest snapshot to make current checkpoint, and delete the obsolete incremental files.
Checklist