feat: use an expiry lru cache for the index backed blockstore

[jacobheun]

Summary

This switches over the dagstore index backed blockstore to use an expiry based LRU cache instead of just an LRU. The existing LRU will never prune unless it fills up and evicts. With the new cache, expired entries will automatically be removed, and in the event the cache hits its max size, the shards closest to expiry will be pruned first.

• Adds a config option, DealmakingConfig.BlockstoreCacheMaxShards to set the max shards in the cache. The previous value was hard coded at 100, which the default has been set to, but we may want to lower this as it can cause process memory to grow quite large. Too low can cause in progress retrievals to fail as shards in use can be pruned.
  • EDIT: reduced to 20 to match default simultaneous retrieval limits
• Adds a config option, DealmakingConfig.BlockstoreCacheExpiry to set the duration a cached shard can be idle for until pruned, default is 10 minutes. Any time the cache is accessed it will be refreshed, so this could be set much lower if memory usage is a problem.
  • EDIT: reduced to 30 seconds

Memory Usage Before

Prior to the change with retrieval load on the system, followed by allowing the system to idle, heap space increases and is not reclaimed.

Looking at a diff of the heap space allocation we can see an increase in the piece reader memory pipeline despite the system being idle.

Memory Usage After

With these changes, allowing the system to idle after retrievals we can see in use heap decrease back to pre load levels.

Performing a diff of heap utilization we can see no significant markers of memory increase...

... and a clear reduction in piece reader heap between time in load and idle time

TODO

• fix ci
• bump dagstore version once released

Dependencies

• requires feat: add expiration support to index backed blockstore dagstore#150

[dirkmc]

Looks good 👍
Feel free to merge once the dagstore release is ready.

[dirkmc]

I'd suggest we decrease these defaults to:

• BlockstoreCacheMaxShards: 16
  16 seems like a reasonable default max for concurrent bitswap retrievals of a piece, and we don't want to go too much higher as each shard can potentially use GiBs of memory
• BlockstoreCacheExpiry: 30s
  In the worst case we need to recreate the blockstore:
  • read the index from disk (about 5MB, should take no more than a few hundred milli-seconds)
  • get a reader over the piece (should take no more than a few hundred milli-seconds)

[jacobheun]

BlockstoreCacheExpiry: 30s should be a reasonable change. It gives plenty of buffer on an active retrieval and can be always be increased.

BlockstoreCacheMaxShards: 16 - Thinking more about this I think we should set this at 20, as it's currently our DefaultSimultaneousTransfers (yes, it only applies to Graphsync but does keep consistency in our defaults). I also think we should adjust the bitswap retrieval defaults to match this, 20.

[dirkmc]

Makes sense, let's go for BlockstoreCacheMaxShards: 20 👍