Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: enable caching when using object store #928

Merged
merged 6 commits into from
Feb 7, 2023

Conversation

e1ijah1
Copy link
Contributor

@e1ijah1 e1ijah1 commented Feb 1, 2023

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

Enable cache layer by default when using s3/OSS storage in the GreptimeDB.

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

closes #918

@codecov
Copy link

codecov bot commented Feb 1, 2023

Codecov Report

Merging #928 (b35d8d3) into develop (9a30ba0) will decrease coverage by 0.55%.
The diff coverage is 60.90%.

@@             Coverage Diff             @@
##           develop     #928      +/-   ##
===========================================
- Coverage    86.14%   85.59%   -0.55%     
===========================================
  Files          434      445      +11     
  Lines        58411    59936    +1525     
===========================================
+ Hits         50316    51305     +989     
- Misses        8095     8631     +536     
Flag Coverage Δ
rust 85.59% <60.90%> (-0.55%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/datanode/src/datanode.rs 57.62% <ø> (ø)
src/datanode/src/instance.rs 41.56% <0.00%> (-16.21%) ⬇️
src/object-store/src/lib.rs 100.00% <ø> (ø)
tests-integration/src/test_util.rs 79.47% <0.00%> (-9.10%) ⬇️
src/object-store/src/cache_policy.rs 90.54% <90.54%> (ø)
src/object-store/src/test_util.rs 0.00% <0.00%> (-100.00%) ⬇️
src/servers/src/query_handler/sql.rs 36.53% <0.00%> (-24.76%) ⬇️
src/frontend/src/instance/standalone.rs 61.70% <0.00%> (-21.16%) ⬇️
src/common/error/src/ext.rs 75.28% <0.00%> (-20.44%) ⬇️
src/servers/src/http/handler.rs 47.82% <0.00%> (-18.85%) ⬇️
... and 112 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

src/datanode/src/instance.rs Outdated Show resolved Hide resolved
src/object-store/src/cache.rs Outdated Show resolved Hide resolved
@killme2008
Copy link
Contributor

Looks great. It's a pity that opendal doesn't support cache eviction by disk or memory quota, the cached objects may consume too much memory or disk. But I think it's okay right now, we still have to design and impl a more powerful cache layer for object storages.

@waynexia
Copy link
Member

waynexia commented Feb 2, 2023

I would prefer to leave memory to a more fine-grained cache manager like page cache or query cache. They are much more expensive than disks and may not provide a comparable performance improvement when used to cache the entire file.

@e1ijah1 e1ijah1 requested review from killme2008 and removed request for waynexia February 5, 2023 09:03
src/datanode/src/instance.rs Outdated Show resolved Hide resolved
src/datanode/src/instance.rs Outdated Show resolved Hide resolved
src/object-store/src/cache_policy.rs Outdated Show resolved Hide resolved
src/object-store/src/cache_policy.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@killme2008 killme2008 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@waynexia waynexia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking great 👍 Thanks @e1ijah1, I only left a nit-picking style comment.

src/object-store/src/cache_policy.rs Outdated Show resolved Hide resolved
@killme2008 killme2008 merged commit 926022e into GreptimeTeam:develop Feb 7, 2023
paomian pushed a commit to paomian/greptimedb that referenced this pull request Oct 19, 2023
* feat: enable caching when using object store

* feat: support file cache for object store

* feat: maintaining the cached files with lru

* fix: improve the code

* empty commit

* improve the code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enable caching in opendal when using s3/oss storage.
3 participants