Implement cache eviction #610
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduce a `CacheLimit` argument for the `DiskDataCache` to configure the eviction policy. If a limit is set, the cache will track usage of the stored blocks. On put, it will check if the limit has been exceeded and start removing the least recently used blocks if required. Signed-off-by: Alessandro Passaro <alexpax@amazon.co.uk>
passaro
temporarily deployed
to
PR integration tests
GitHub Actions
Inactive
passaro
temporarily deployed
to
PR integration tests
GitHub Actions
Inactive
passaro
temporarily deployed
to
PR integration tests
GitHub Actions
Inactive
passaro
temporarily deployed
to
PR integration tests
GitHub Actions
Inactive
dannycjones
reviewed
Nov 16, 2023
passaro
had a problem deploying
to
PR integration tests
GitHub Actions
Failure
passaro
had a problem deploying
to
PR integration tests
GitHub Actions
Failure
passaro
had a problem deploying
to
PR integration tests
GitHub Actions
Failure
passaro
had a problem deploying
to
PR integration tests
GitHub Actions
Failure
Signed-off-by: Alessandro Passaro <alexpax@amazon.co.uk>
passaro
force-pushed
the
cache-eviction
branch
from
127a27d to
b07256c
Compare
passaro
temporarily deployed
to
PR integration tests
GitHub Actions
Inactive
passaro
temporarily deployed
to
PR integration tests
GitHub Actions
Inactive
passaro
temporarily deployed
to
PR integration tests
GitHub Actions
Inactive
passaro
temporarily deployed
to
PR integration tests
GitHub Actions
Inactive
dannycjones
previously approved these changes
Nov 16, 2023
Comment on lines
+247
to
+250
| Ok(stats) if stats.total_space() == 0 => { | ||
| warn!("unable to determine available space"); | ||
| return false; | ||
| } |
There was a problem hiding this comment.
Thinking about this a little more, maybe we should switch the cache limit to Unbounded when we encounter this... otherwise this is a lot of WARN logging 😄
Though that requires us to switch to &mut self or something. Maybe let's switch to unbounded if we notice that we can't determine total space?
(I'm happy for this to be follow-up, we're not targeting filesystems that lie about fsstat.)
Signed-off-by: Alessandro Passaro <alexpax@amazon.co.uk>
passaro
temporarily deployed
to
PR integration tests
GitHub Actions
Inactive
passaro
temporarily deployed
to
PR integration tests
GitHub Actions
Inactive
passaro
temporarily deployed
to
PR integration tests
GitHub Actions
Inactive
passaro
temporarily deployed
to
PR integration tests
GitHub Actions
Inactive
dannycjones
approved these changes
Nov 16, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of change
Introduce a
CacheLimitargument for theDiskDataCacheto configure the eviction policy. If a limit is set, the cache will track usage of the stored blocks. On put, it will check if the limit has been exceeded and start removing the least recently used blocks if required.Relevant issues: #255
Does this change impact existing behavior?
No.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the Developer Certificate of Origin (DCO).