Amend Blobs Database to Use a Rotating Buffer for Keys

[rauljordan]

What type of PR is this?

Feature

What does this PR do? Why is it needed?

Bolt DB does not play nicely with pruning, as evidenced by extensive pains developing slasher. Blob data can be huge, and we will be storing a lot of it in Prysm with EIP-4844. Instead of having to prune, we devise a database schema for blobs where keys are a rotating buffer that wraps around once the expiration period for blobs is reached. For example, imagine we store 2 epochs max worth of blobs. Then, once we reach epoch 2, we will be overriding the contents of epoch 0 in the database by "wrapping around" the keys using a modulo operation. This removes the need for any kind of deletion routine or frequent pruning and keeps the database size bounded.

There is some complexity involved, as we there could be multiple blob sidecars per slot, so we can't do a naive overwriting algorithm. Instead, here's how the saving of blobs works:

When we receive a blob:

1. Convert slot using a modulo operator to [0, maxSlots] where maxSlots = MAX_BLOB_EPOCHS*SLOTS_PER_EPOCH
2. Compute key for blob as bytes(slot_to_rotating_buffer(blob.slot)) ++ bytes(blob.slot) ++ blob.block_root
3. Begin the save algorithm: If the incoming blob has a slot bigger than the last saved slot at the spot in the rotating keys buffer, we overwrite all elements.

firstElemKey = getFirstElement(bucket)
shouldOverwrite = blob.slot > bytes_to_slot(firstElemKey[8:16])
if shouldOverwrite:
for existingKey := seek prefix bytes(slot_to_rotating_buffer(blob.slot))
  bucket.delete(existingKey)
bucket.put(key, blob)

The unit test included shows exactly how the algorithm works.

[rkapka]

Should this be configurable via a flag? I can imagine users wanting to save blobs for longer.

[rauljordan]

Yep it should be done via a feature config IMO...will reserve that for a second PR

[rkapka]

Is this even possible?

[rauljordan]

Yeah maybe someone changes the schema. It's just a defensive check to prevent panics and instead error gracefully because otherwise we'll be accessing slice indices out of range

[rkapka]

Aren't we possibly leaving the db in an inconsistent state?

[rauljordan]

Yeah good point, might instead collect the errors

[rkapka]

Is this even possible?

[rauljordan]

Same comment as above

[rkapka]

You should not append 64 because the original sidecar at round buffer's 0 value had a different slot.

[rauljordan]

In this example, the key we are expecting is 0 ++ 64 ++ bytes32(foo-64). A key contains the round buffer, the slot, and the block root. The new value we are storing at round buffer 0 will be under this new key, so this is the correct value to expect

[terencechain]

Lgtm, let's review this more in detail when it goes to the upstream develop branch

[rkapka]

I still argue that TestBlobsSidecar_Overwriting does not test overwriting. The last assertion will pass even if old blobs were not overwritten. We should check the number of items under key prefix bytesutil.SlotToBytesBigEndian(0).

[terencechain]

@rkapka I'm going to move this to prysm branch, we'll make a note of your comment there. Thanks!