Splitstore Enhanchements #6474

this is necessary to avoid wearing clown shoes when the node stays offline for an extended period of time (more than 1 finality). Basically it gets quite slow if we do the full 2 finality walk, so we try to avoid it unless necessary. The conditions under which a full walk is necessary is if there is a sync gap (most likely because the node was offline) during which the tracking of writes is inaccurate because we have not yet delivered the HeadChange notification. In this case, it is possible to have actually hot blocks to be tracked before the boundary and fail to mark them accordingly. So when we detect a sync gap, we do the full walk; if there is no sync gap, we can just use the much faster boundary epoch walk.

for maximal safety.

The issue: head change notifications are not emitted until after catching up, which results in all writes during a catch up period being tracked at the base epoch.

to account for off by one conditions

get rid of the CompactionCold construct, run a single compaction on catch up

otherwise we'll wear clown shoes with the slack and end up in continuous compaction.

It is possible for an object to be written or recreated (and checked with Has) after the mark completes and during the purge; if this happens we will purge a live block.

- allocate once - log purge count

otherwise the walk itself precludes purge... duh!

Rationale: the VM uses the Has check to avoid issuing a duplicate Write in the blockstore. This means that live objects that would be otherwise written are not actually written, resulting in the first write epoch being considered the write epoch.

bolt performance leaves something to be desired; doing a single Put takes 10ms, about the same time as batching thousands of them.

… avoid races

genesis is written (!) before starting the splitstore, so curTs is nil

avoid crawling everything to a halt

This cause Has to return false if it fails to traverse/protect all links, which would cause the vm to recompute.

it is a noop but it still takes (a lot of) time because it has to read all the cold blocks.

…ent walking

stack tracing is slow.

cid copies the bytes so it's safe

- remove defunct tracking store implementations - update splitstore node config - use mark set type config option (defaulting to mapts); a memory constrained node may want to use an on-disk one

…erences

so that we can't shoot ourselves in the foot by deleting the constituents of a DAG while it is still in the hotstore.

Now that we delete objects heaviest first, we don't have to do deep walk and rescan gymnastics.

Just the purge; the rest is not critical -- e.g. it's ok if we do some duplicate copies to the coldstore, we'll have gc soon.

so that we run compaction every finality, once we've first compacted

seems that something is writing DAGs before its consituents, which causes problems.

…bjects

they don't make it to the blockstore anyway

those missing objects don't seem to ever get there, are they from an abandoned fork?

so that we wait 6min at most, not 12.

really, it's concurrent marking and there is no reason to have two different marksets

it's a false positive, function doesn't escape.

cids that come out of the hotstore with ForEach are raw.

otherwise the root object will be raw, but internal references will be dag; duplicate work.

so as to avoid making a gazillion of strings

it's taking a long time to compute weights...

sort is still taking a long time, this should be as fast as it gets.

a concurrent marking can panic.

so that we have a reasonably quick graceful shutdown

so that we don't do it too often and also cover warmup.

Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>

As pointed out by magik, it is possible to deadlock if the view callback performs a blockstore operation while a Lock is pending. This fixes the issue by optimistically tracking the reference before actually calling the underlying View and limiting the scope of the lock.

this has very noticeable impact in initial marking time; it also allows us to get rid of the confusing ts monikers.

…d to do concurrent marking The problem is that it is possible that an inline marking might take minutes for some objects (infrequent, but still possible for state roots and prohibitive if that's a block validation). So we simply track references continuously and rely on the main compaction thread to trigger concurrent marking for all references at opportune moments. Assumption: we can mark references faster than they are created during purge or else we'll never purge anything.

so that we preclude the following scenario: Start compaction. Start view. Finish compaction. Start compaction. which would not wait for the view to complete.

Because stebalien has allergies.

We can add after Wait is called, which is problematic with WaitGroups. This instead uses a mx/cond combo and waits while the count is > 0. The only downside is that we might needlessly wait for (a bunch) of views that started while the txn is active, but we can live with that.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Splitstore Enhanchements #6474

Splitstore Enhanchements #6474

Commits on Jul 4, 2021

Commits on Jul 5, 2021

Commits on Jul 6, 2021

Commits on Jul 7, 2021

Commits on Jul 8, 2021

Commits on Jul 9, 2021

Commits on Jul 10, 2021

Commits on Jul 13, 2021