-
Notifications
You must be signed in to change notification settings - Fork 308
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
build(deps): Upgrade OpenDAL to 0.47 #4224
Conversation
Signed-off-by: tison <wander4096@gmail.com>
This comment was marked as spam.
This comment was marked as spam.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files ignored due to path filters (1)
Cargo.lock
is excluded by!**/*.lock
Files selected for processing (12)
- src/common/datasource/src/file_format.rs (1 hunks)
- src/common/datasource/src/file_format/csv.rs (1 hunks)
- src/common/datasource/src/file_format/json.rs (1 hunks)
- src/common/datasource/src/file_format/parquet.rs (2 hunks)
- src/mito2/src/cache/write_cache.rs (1 hunks)
- src/mito2/src/sst/index/applier.rs (2 hunks)
- src/mito2/src/sst/index/store.rs (1 hunks)
- src/object-store/Cargo.toml (1 hunks)
- src/object-store/src/layers/lru_cache.rs (2 hunks)
- src/object-store/src/layers/lru_cache/read_cache.rs (8 hunks)
- src/object-store/src/layers/prometheus.rs (2 hunks)
- src/operator/src/statement/copy_table_from.rs (2 hunks)
Files skipped from review due to trivial changes (1)
- src/object-store/src/layers/prometheus.rs
Additional comments not posted (11)
src/object-store/Cargo.toml (1)
20-20
: Dependency version updated successfully.The
opendal
dependency has been updated to version "0.47". Ensure that all features listed under this dependency are compatible with the new version.src/common/datasource/src/file_format.rs (1)
152-154
: Enhanced asynchronous handling inopen_with_decoder
.The function now properly handles asynchronous operations with
.await
and improved error mapping. This aligns with Rust's best practices for asynchronous programming and error handling.src/common/datasource/src/file_format/json.rs (1)
90-97
: Improved error handling inJsonFormat
.The addition of
.context(error::ReadObjectSnafu { path })?
after asynchronous reader operations provides better error context, enhancing maintainability and debuggability of the code.src/common/datasource/src/file_format/parquet.rs (1)
55-62
: Consistent and robust error handling inParquetFormat
.The addition of
.context(error::ReadObjectSnafu { path })?
after asynchronous operations is a significant enhancement for error handling, ensuring more reliable and maintainable code.src/mito2/src/sst/index/store.rs (1)
70-72
: Enhanced asynchronous operation and error handling inreader
method.The changes introduce proper awaiting of asynchronous operations and improve error handling by using
.context(OpenDalSnafu)
. This ensures that any errors during file operations are caught and handled appropriately, enhancing the robustness of the method.src/mito2/src/sst/index/applier.rs (1)
128-140
: Refined asynchronous read operation incached_puffin_reader
.The changes here enhance the asynchronous read operation by properly awaiting the operation and adding context-specific error handling. This improvement is crucial for robust error reporting and handling, ensuring that errors are not only caught but also described with sufficient context.
src/common/datasource/src/file_format/csv.rs (1)
172-179
: Improved asynchronous operation and error handling ininfer_schema
.The changes introduce proper awaiting of asynchronous operations and enhance error handling using
.context(error::ReadObjectSnafu)
. This ensures that any errors during file operations are caught and handled appropriately, enhancing the robustness and reliability of the schema inference process.src/object-store/src/layers/lru_cache/read_cache.rs (1)
164-186
: Refined read operation handling inread
method.The changes here improve how read operations are handled by properly awaiting asynchronous operations and introducing a boxed reader, which is crucial for handling polymorphism in Rust effectively. This ensures that the read operation is not only asynchronous but also adaptable to different reader implementations.
src/mito2/src/cache/write_cache.rs (1)
191-193
: Enhanced asynchronous read and error handling inupload
method.The changes introduce proper awaiting of asynchronous operations and enhance error handling using
.context(error::OpenDalSnafu)
. This ensures that any errors during file operations are caught and handled appropriately, enhancing the robustness of the upload process.src/operator/src/statement/copy_table_from.rs (2)
159-160
: Enhanced error handling in asynchronous file reading operations.The addition of
.await
and.context
for error handling in thecollect_metadata
method under theParquet
format case is a significant improvement. This ensures that any errors during the file reading operation are properly handled and contextualized, which aligns with the PR's objectives to enhance error handling mechanisms.
306-307
: Consistent application of enhanced error handling in asynchronous operations.Similar to the previous comment, the use of
.await
and.context
in thebuild_read_stream
method for theParquet
case is commendable. This consistency in applying enhanced error handling across different parts of the file ensures that the system's robustness and reliability are maintained, especially when dealing with asynchronous operations.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (1)
- src/mito2/src/sst/index/applier.rs (2 hunks)
Files skipped from review as they are similar to previous changes (1)
- src/mito2/src/sst/index/applier.rs
The experience of implementing caching in OpenDAL layer is not cool. I plan to make it easier in the future. |
* Fix cache for 0471 Signed-off-by: Xuanwo <github@xuanwo.io> * Make clippy happy Signed-off-by: Xuanwo <github@xuanwo.io> --------- Signed-off-by: Xuanwo <github@xuanwo.io>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (4)
- src/datanode/src/store.rs (3 hunks)
- src/object-store/src/layers/lru_cache.rs (4 hunks)
- src/object-store/src/layers/lru_cache/read_cache.rs (7 hunks)
- src/object-store/tests/object_store_test.rs (4 hunks)
Additional comments not posted (19)
src/object-store/src/layers/lru_cache.rs (3)
Line range hint
36-51
: Ensure correctness of async operations and error handling.The
new
function initializes theLruCacheLayer
with aReadCache
and then recovers the cache. Verify the correctness of the async operations and error handling.
Line range hint
78-119
: Ensure correctness and thread safety of async and blocking operations.The
LayeredAccess
implementation forLruCacheAccess
defines various async and blocking operations. Verify the correctness and thread safety of these operations.
Line range hint
59-65
: Ensure correctness of cloning operation.The
Layer
implementation forLruCacheLayer
defines theLayeredAccess
type and thelayer
function. Verify the correctness of the cloning operation.src/datanode/src/store.rs (2)
Line range hint
32-44
: Ensure correctness of async operations and error handling.The
new_object_store
function initializes an object store based on the provided configuration and data home. Verify the correctness of the async operations and error handling.Verification successful
Line range hint
32-44
: Verified async operations and error handling.The
new_object_store
function correctly handles async operations usingawait
and ensures proper error handling withcontext
.
- Verified usage of
await
for async operations.- Verified usage of
context
for error handling.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the correctness of async operations and error handling in the `new_object_store` function. # Test: Check for the usage of `await` and `context` in the `new_object_store` function. rg 'await' src/datanode/src/store.rs rg 'context' src/datanode/src/store.rsLength of output: 854
112-114
: Ensure correctness of async operations and error handling.The
create_object_store_with_cache
function initializes a cache layer for the object store based on the provided configuration. Verify the correctness of the async operations and error handling.Verification successful
Correctness of async operations and error handling verified.
The
create_object_store_with_cache
function correctly uses async operations and handles errors as expected.
- The function is defined as
async fn
.- It uses
await
for async operations.- Error handling is done using the
context
method.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the correctness of async operations and error handling in the `create_object_store_with_cache` function. # Test: Check for the usage of `await` and `context` in the `create_object_store_with_cache` function. rg 'await' src/datanode/src/store.rs rg 'context' src/datanode/src/store.rsLength of output: 854
Script:
#!/bin/bash # Extract the implementation of the `create_object_store_with_cache` function. ast-grep --lang rust --pattern 'async fn create_object_store_with_cache($_) -> $_ { $$$ }' src/datanode/src/store.rsLength of output: 118
Script:
#!/bin/bash # Broaden the search to include both async and non-async functions for `create_object_store_with_cache`. rg 'fn create_object_store_with_cache' src/datanode/src/store.rsLength of output: 107
Script:
#!/bin/bash # Extract the complete implementation of the `create_object_store_with_cache` function. rg -A 20 'async fn create_object_store_with_cache' src/datanode/src/store.rsLength of output: 875
Script:
#!/bin/bash # Extract more lines to get the complete implementation of the `create_object_store_with_cache` function. rg -A 50 'async fn create_object_store_with_cache' src/datanode/src/store.rsLength of output: 2017
Script:
#!/bin/bash # Extract more lines to get the complete implementation of the `create_object_store_with_cache` function. rg -A 100 'async fn create_object_store_with_cache' src/datanode/src/store.rsLength of output: 3721
src/object-store/src/layers/lru_cache/read_cache.rs (5)
Line range hint
81-252
: Ensure correctness of async operations and error handling.The
ReadCache
implementation defines various async operations for cache management. Verify the correctness of these operations and error handling.
72-79
: Ensure correctness of cloning operation.The
Clone
implementation forReadCache
clones the file cache and memory cache. Verify the correctness of the cloning operation.
Line range hint
302-312
: Ensure correctness of test cases.The
test_can_cache
function tests thecan_cache
function. Verify the correctness of the test cases.Verification successful
Ensure correctness of test cases.
The test cases in the
test_can_cache
function are correct and consistent with the implementation of thecan_cache
function.
- The function
can_cache
returnstrue
if the path does not end with_last_checkpoint
.- The test cases correctly cover both scenarios where the path ends and does not end with
_last_checkpoint
.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the correctness of test cases in the `test_can_cache` function. # Test: Check for the usage of `assert!` in the `test_can_cache` function. rg 'assert!' src/object-store/src/layers/lru_cache/read_cache.rsLength of output: 394
Script:
#!/bin/bash # Locate the implementation of the `can_cache` function to verify its correctness. ast-grep --lang rust --pattern 'fn can_cache($_) { $$$ }'Length of output: 59
Script:
#!/bin/bash # Broaden search to locate the `can_cache` function across all Rust files. rg 'fn can_cache' --type rustLength of output: 118
Script:
#!/bin/bash # Extract the implementation of the `can_cache` function from the identified file. ast-grep --lang rust --pattern 'fn can_cache($_) { $$$ }' src/object-store/src/layers/lru_cache/read_cache.rsLength of output: 111
Script:
#!/bin/bash # Extract the implementation of the `can_cache` function from the identified file using sed. sed -n '/fn can_cache/,/}/p' src/object-store/src/layers/lru_cache/read_cache.rsLength of output: 198
55-60
: Ensure correctness of cache key generation.The
read_cache_key
function generates a unique cache key for the read path and range. Verify the correctness of the key generation.
298-299
: Ensure correctness of reader conversion.The
to_output_reader
function converts the input reader to the output reader. Verify the correctness of the conversion.Verification successful
Verification successful.
The conversion in the
to_output_reader
function is correct. The type aliasReader
is defined asBox<dyn ReadDyn>
, making the use ofBox::new(input.1)
appropriate.
- The
to_output_reader
function takes a tuple(RpRead, R)
and returns(RpRead, Reader)
.- The second element of the tuple,
input.1
, is boxed into aBox<dyn ReadDyn>
.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the correctness of reader conversion in the `to_output_reader` function. # Test: Check for the usage of `Box::new` in the `to_output_reader` function. rg 'Box::new' src/object-store/src/layers/lru_cache/read_cache.rsLength of output: 98
Script:
#!/bin/bash # Description: Extract the types of `input.0` and `input.1` in the `to_output_reader` function to verify the correctness of the conversion. # Locate the `to_output_reader` function and extract its definition along with the types of `input.0` and `input.1`. ast-grep --lang rust --pattern $'fn to_output_reader($_: ($_, $_)) -> $_ { $$$ }' src/object-store/src/layers/lru_cache/read_cache.rsLength of output: 136
Script:
#!/bin/bash # Description: Locate the `to_output_reader` function definition and extract its details to verify the correctness of the conversion. # Search for the `to_output_reader` function definition in the specified file. rg 'fn to_output_reader' src/object-store/src/layers/lru_cache/read_cache.rs -A 10Length of output: 319
Script:
#!/bin/bash # Description: Confirm the type of `Reader` to ensure the correctness of the conversion in the `to_output_reader` function. # Search for the definition or type alias of `Reader` in the specified file. rg 'type Reader' src/object-store/src/layers/lru_cache/read_cache.rsLength of output: 70
Script:
#!/bin/bash # Description: Confirm the type of `Reader` to ensure the correctness of the conversion in the `to_output_reader` function. # Search for the definition or type alias of `Reader` in the entire repository. rg 'type Reader' -t rustLength of output: 946
src/object-store/tests/object_store_test.rs (9)
25-25
: ImportList
andRead
fromopendal::raw::oio
.This change aligns with the updated
opendal
dependency.
26-26
: ImportAccess
,OpList
, andOpRead
fromopendal::raw
.This change aligns with the updated
opendal
dependency.
232-232
: Usebuilder.build()
instead ofObjectStore::new(builder).unwrap().finish()
.This change aligns with the updated
opendal
dependency and simplifies the code.
240-240
: Usebuilder.build()
instead ofObjectStore::new(builder).unwrap().finish()
.This change aligns with the updated
opendal
dependency and simplifies the code.
245-247
: UseOperatorBuilder
to create thestore
withcache_layer
.This change aligns with the updated
opendal
dependency and improves the readability and maintainability of the code.
257-257
: Addassert_lru_cache
utility function.This function enhances the test coverage for the LRU cache functionality.
263-286
: Addassert_cache_files
utility function.This function enhances the test coverage for the LRU cache functionality.
315-315
: Usefile_cache.clone()
instead offile_cache
.This change aligns with the updated
opendal
dependency and simplifies the code.
315-315
: Usefile_cache.clone()
instead offile_cache
.This change aligns with the updated
opendal
dependency and simplifies the code.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4224 +/- ##
==========================================
- Coverage 84.84% 84.55% -0.29%
==========================================
Files 1049 1049
Lines 186267 186312 +45
==========================================
- Hits 158029 157541 -488
- Misses 28238 28771 +533 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files ignored due to path filters (1)
Cargo.lock
is excluded by!**/*.lock
Files selected for processing (2)
- src/datanode/src/store.rs (3 hunks)
- src/object-store/src/layers/lru_cache/read_cache.rs (7 hunks)
Files skipped from review as they are similar to previous changes (2)
- src/datanode/src/store.rs
- src/object-store/src/layers/lru_cache/read_cache.rs
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Signed-off-by: tison <wander4096@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files ignored due to path filters (1)
Cargo.lock
is excluded by!**/*.lock
Files selected for processing (2)
- src/object-store/src/layers/lru_cache.rs (4 hunks)
- src/object-store/src/layers/lru_cache/read_cache.rs (7 hunks)
Files skipped from review as they are similar to previous changes (2)
- src/object-store/src/layers/lru_cache.rs
- src/object-store/src/layers/lru_cache/read_cache.rs
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (2)
- src/object-store/src/layers/lru_cache.rs (4 hunks)
- src/object-store/src/layers/lru_cache/read_cache.rs (7 hunks)
Files skipped from review as they are similar to previous changes (1)
- src/object-store/src/layers/lru_cache.rs
Additional comments not posted (10)
src/object-store/src/layers/lru_cache/read_cache.rs (10)
55-60
: LGTM!The function
read_cache_key
now usesOpRead
for generating the cache key. This change is consistent with the new structure.
65-67
: LGTM!The
ReadCache
struct is now generic over a typeC
that implements theAccess
trait. TheClone
implementation has been added correctly. No issues found.Also applies to: 72-79
Line range hint
81-109
: LGTM!The
new
function now accepts anArc<C>
instead of anOperator
. The changes are consistent and correctly implemented.
120-123
: LGTM!The
stat
method has been renamed tocache_stat
. The changes are consistent and correctly implemented.
Line range hint
145-165
: LGTM!The
recover_cache
method now usesOpList
andOpStat
. The changes are consistent and correctly implemented.
178-231
: LGTM!The
read_from_cache
method now returns aBox<dyn ReadDyn>
instead ofArc<dyn ReadDyn>
. The changes are consistent and correctly implemented.
234-252
: LGTM!The
try_write_cache
method now accepts aReader
instead of aBuffer
. The changes are consistent and correctly implemented.
Line range hint
254-294
: LGTM!The
read_remote
method now accepts aReader
instead of aBuffer
. The changes are consistent and correctly implemented.
298-299
: LGTM!The function
to_output_reader
has been updated to return aBox<dyn ReadDyn>
instead of anArc<dyn ReadDyn>
. This change is consistent with the new structure.
Line range hint
303-312
: LGTM!The test cases for the
can_cache
function cover various scenarios and are correctly implemented.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files ignored due to path filters (1)
Cargo.lock
is excluded by!**/*.lock
Files selected for processing (1)
- src/object-store/src/layers/lru_cache.rs (4 hunks)
Files skipped from review as they are similar to previous changes (1)
- src/object-store/src/layers/lru_cache.rs
* catch up changes Signed-off-by: tison <wander4096@gmail.com> * fmt Signed-off-by: tison <wander4096@gmail.com> * Fix cache for 0471 (GreptimeTeam#7) * Fix cache for 0471 Signed-off-by: Xuanwo <github@xuanwo.io> * Make clippy happy Signed-off-by: Xuanwo <github@xuanwo.io> --------- Signed-off-by: Xuanwo <github@xuanwo.io> * tidy Signed-off-by: tison <wander4096@gmail.com> * use opendal's exported type Signed-off-by: tison <wander4096@gmail.com> * clippy Signed-off-by: tison <wander4096@gmail.com> * fmt Signed-off-by: tison <wander4096@gmail.com> --------- Signed-off-by: tison <wander4096@gmail.com> Signed-off-by: Xuanwo <github@xuanwo.io> Co-authored-by: Xuanwo <github@xuanwo.io>
I hereby agree to the terms of the GreptimeDB CLA.
Refer to a related PR or issue link (optional)
What's changed and what's your intention?
Upgrade OpenDAL to 0.47.1.
Checklist
Summary by CodeRabbit
LruCacheLayer
andLruCacheAccess
generic over a type that implementsAccess
.read_cache_stat
tocache_stat
for consistency.