Search local shards directly to find shard to evict #791
Conversation
|
https://github.com/databricks-mosaic/regression-testing/actions/runs/11078601583 Adding regression runs here^ |
|
You can invoke like this yourself to test against cache_limit yaml! 
Results will show up in #genai-regression-testing-dev in a couple hours |
Throughput looks normal, can we rev/merge @snarayan21 @XiaohanZhangCMU |
There was a problem hiding this comment.
looks nice! agreed with @XiaohanZhangCMU 's comments. I don't think we actually need a mechanism to make sure the local shard isn't used...according to regression tests, this seems fine to me. Also we're choosing the coldest shard anyways.
|
@snarayan21 yeah, I'm ok with it. The question I had is not valid, which should have been handled by the get_item retry logic already. |
|
Closing because it is superseded by 795. |
This PR changes the search for coldest shard to avoid looping over remote shards by considering local shards only as possible candidates for eviction.