Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(s3stream/wal): group commit #422

Merged
merged 17 commits into from
Oct 23, 2023
Merged

feat(s3stream/wal): group commit #422

merged 17 commits into from
Oct 23, 2023

Conversation

Chillax-0v0
Copy link
Contributor

No description provided.

@Chillax-0v0
refactor: WriteRecordTask#future -> WriteBlockTask#futures
4bbabc9 
Signed-off-by: Ning Yu <ningyu@automq.com>
@Chillax-0v0
refactor: use common utils to new thread pools
1b3dc16 
Signed-off-by: Ning Yu <ningyu@automq.com>
@Chillax-0v0
refactor: merge header and body into record in WriteBlockTask
3fefa90 
Signed-off-by: Ning Yu <ningyu@automq.com>
@Chillax-0v0
test: use flushed offset to trim
ad477fe 
Signed-off-by: Ning Yu <ningyu@automq.com>
@Chillax-0v0
feat: WriteBlockTask#addRecord
29154be 
Signed-off-by: Ning Yu <ningyu@automq.com>
@Chillax-0v0
feat: SlidingWindowService#newBlock
57ac017 
Signed-off-by: Ning Yu <ningyu@automq.com>
@Chillax-0v0
refactor: move Flusher from WriteBlockTask to SlidingWindowService
1b3ebb1 
Signed-off-by: Ning Yu <ningyu@automq.com>
@Chillax-0v0
refactor: change signature of WriteBlockTask#addrecord
5fc3801 
Signed-off-by: Ning Yu <ningyu@automq.com>
@Chillax-0v0
fix: init the currentWriteTask
570621f 
Signed-off-by: Ning Yu <ningyu@automq.com>
@Chillax-0v0
fix: update the next write offset when shutdown gracefully
ca18561 
Signed-off-by: Ning Yu <ningyu@automq.com>
@Chillax-0v0
feat: a semaphore for io thread pool
9855ab3 
Signed-off-by: Ning Yu <ningyu@automq.com>
@Chillax-0v0
feat: calculate start offset by blocks
1fbfd1e 
Signed-off-by: Ning Yu <ningyu@automq.com>
@Chillax-0v0
refactor: rename WriteBlockTask to Block
0990863 
Signed-off-by: Ning Yu <ningyu@automq.com>
@Chillax-0v0
feat: try to poll next block after writing the previous one
470f65e 
Signed-off-by: Ning Yu <ningyu@automq.com>
@Chillax-0v0
fix: reset the position of BlockImpl#data
2eb518f 
Signed-off-by: Ning Yu <ningyu@automq.com>
@Chillax-0v0
fix: calculate the crc of record body
1213c0a 
Signed-off-by: Ning Yu <ningyu@automq.com>
@Chillax-0v0 Chillax-0v0 added s3stream wal the WAL of S3 stream labels Oct 20, 2023
@Chillax-0v0 Chillax-0v0 self-assigned this Oct 20, 2023
@codecov
Copy link

codecov bot commented Oct 20, 2023 •
edited
Loading

Codecov Report

Merging #422 (d0fd5a6) into main (a1ab50f) will increase coverage by 1.27%.
Report is 9 commits behind head on main.
The diff coverage is n/a.

Additional details and impacted files 
@@             Coverage Diff              @@
##               main     #422      +/-   ##
============================================
+ Coverage     58.79%   60.07%   +1.27%     
- Complexity      773      792      +19     
============================================
  Files            94       94              
  Lines          5655     5686      +31     
  Branches        525      528       +3     
============================================
+ Hits           3325     3416      +91     
+ Misses         2014     1951      -63     
- Partials        316      319       +3
Components Coverage Δ
RocketMQ Broker ∅ <ø> (∅)
RocketMQ Common 12.50% <ø> (ø)
RocketMQ Controller 59.18% <ø> (+0.39%) ⬆️
RocketMQ Proxy 44.63% <ø> (+5.77%) ⬆️
RocketMQ Store 74.45% <ø> (-0.09%) ⬇️
RocketMQ Stream ∅ <ø> (∅)
RocketMQ Metadata 18.00% <ø> (ø)

see 13 files with indirect coverage changes

@github-actions
Copy link

github-actions bot commented Oct 20, 2023 •
edited
Loading

Qodana for JVM

3 new problems were found

Inspection name Severity Problems
Class is exposed outside of its visibility scope 🔶 Warning 2
Declaration has problems in Javadoc references 🔶 Warning 1

💡 Qodana analysis was run in the pull request mode: only the changed files were checked
☁️ View the detailed Qodana report

Contact Qodana team

Contact us at qodana-support@jetbrains.com

superhx
superhx reviewed Oct 23, 2023
View reviewed changes
s3stream/src/main/java/com/automq/stream/s3/wal/BlockImpl.java
// TODO use composite buffer
if (requiredCapacity > data.capacity()) {
int newCapacity = Math.max(data.capacity() * 2, (int) requiredCapacity);
ByteBuffer newData = ByteBuffer.allocate(newCapacity);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use DirectByteBufAlloc to alloc and free bytebuf

@Chillax-0v0
Copy link
Contributor Author

to #173
fix AutoMQ/automq#377

This was linked to issues Oct 23, 2023
WAL throw unexpected exception cause stream offset not continuous AutoMQ/automq#377
Closed
[WAL] Add group commit feature to reduce the number of IO operations #173
Closed
@Chillax-0v0
perf: return an illegal value rather than BlockFullException
d0fd5a6 
Signed-off-by: Ning Yu <ningyu@automq.com>
@superhx superhx merged commit 0f0cc4a into main Oct 23, 2023
9 checks passed
@superhx superhx deleted the feat-s3-stream-wal-group-commit branch October 23, 2023 02:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@superhx superhx superhx approved these changes

@SCNieh SCNieh Awaiting requested review from SCNieh

@daniel-y daniel-y Awaiting requested review from daniel-y

Assignees

@Chillax-0v0 Chillax-0v0

Labels
s3stream wal the WAL of S3 stream
Projects
None yet
Milestone
No milestone
2 participants
@Chillax-0v0 @superhx