Improve mindreader/merger behavior when syncing a large chain from block 1

[sduchesneau]

Context

Currently, the mindreader creates "one-block-files" and the merger collects them to create "merged (100) blocks files".

The reason for this two-steps architecture is for having a few mindreader processes running on HEAD and getting different versions of same block numbers when microforks are created, we want to make sure that any single block (forked out or not) that has gone through our system (mindreader->relayer->...) gets written to a merged blocks file so it can be retrieved afterwards.

This process is NOT MEANT for going over millions of blocks in an attempt to sync/bootstrap a large chain, it is meant for providing a stronger guarantee to the clients accessing the service that any block served close to the HEAD, in real-time, will be able to be served later, even if it was forked out, so the cursors are always valid. When default settings are used to sync linearly a large chain like eos-mainnet, something is bound to fail (usually an out-of-memory error happens at some point in time and crashes the mindreader's nodeos instance into a corrupted state)

This has been discussed in #26 and on multiple other occasions.

However, we keep seeing that users try to sync the whole chain from the start directly (ex: #80) . We talked to some of our users who prefer this approach without going into the operational steps of gathering nodeos snapshots and running them in parallel, even if it takes weeks of sync time. (throwing "time" at the problem instead of operational complexity.).

Proposed behavior

1. Make the mindreader able to decide to produce merged blocks-files depending when it is "catching up" (the timestamp of blocks being processed could be enough to determine that)
2. Make the mindreader able to flush to disk a "partially-merged" blocks-file and append to it on restart.
3. add a dfuseeos sync command that launches only the syncing components:

• mindreader with auto-merge mode
• trxdb-loader
• search-indexer

Note that search-indexer uses small shards (200-blocks) by default, this will be ridiculously inefficient if it covers 128 million blocks, but does not work well with large shards close to the head. However, search-archive does not support serving different sizes of shards directly, so it cannot be run in the same single-process afterwards. A solution will have to be proposed to the user.

[abourget]

About point 3 of Proposed behaviors:

I'd prefer we do not push to users phases of procesing, additional config files, and templates that force them to understand ALL the complexities when they barely are getting started.

I'd prefer the thing just WORKS, and the default behavior is you have to WAIT.

We can have the programs self adjust when they know their status, wait before starting, etc.. based on catch-up mode.

--

Regarding huge indexes, the search-archive, defaults could be a moving window of 1 week or so, so search-archive, knowing its not near live, would just do nothing.

Eventually, you could start from a snapshot, and get a partial sync.. but we'll slowly get there.

[matthewdarwin]

Point (1) on its own without any of the other features is very useful.

[sduchesneau]

flag could be like this: mergeBlockFiles=[always, auto, none]
also add a reference to a "blockmeta" service that may (or may not) be able to

state is [catching up] if:

1. the blocks are at least 5 minutes late, and the destination merged file already exists
2. the blocks are at least 5 minutes late, and a connection to a "blockmeta" service tells us that we are processing blocks that are passed the LIB
3. the blocks are at least 12 hours late

[sduchesneau]

changed a bit the requirements for simplicity:
state is 'catching up' if:

1. the destination merged-block exist (ex: you have a few mindreaders and a merger, one mindreader crashed and restarted from snapshot...)
2. the blocks being processed are at least [x hours] old (12h by default, it is risky to cover 'live' blocks like this, because you could end up with the wrong chain in case there is a huge reorg and the LIB does not advance in an EOS chain)

[sduchesneau]

Modified the requirements again, the "destination merged-block exists" was too risky close to the head..

so now, it will produce merged block files automatically if either of these is true when the mindreader stars:

1. the blocks being processed are at least [x hours] old (12h by default, it is risky to cover 'live' blocks like this, because you could end up with the wrong chain in case there is a huge reorg and the LIB does not advance in an EOS chain)
2. The mindreader can connect to 'blockmeta' and determine the LIB. The current block num + 100 is behind the LIB (so no risk of fork)

When those two conditions become false (when close to HEAD), it will stop producing merged-blocks and go back to producing one-block-files, being careful to change its operation mode only on boundary blocks (modulo 100)

[sduchesneau]

Note that the merger, if it is waiting to produce merged blocks [200,201...] it looks at the merged-blocks-storage and if it sees that file appear, it skips to the next, so it should operate well with the new mindreader behavior.

This removes the need for: a) a "catchup phase" with a specific mindreader flag and b) going through hoops to join the batch-produced blocks with the one produced from the merger.

[abourget]

When it skips, are there no chances of missing forks? How does it know what was in that file in order to skip to the right place, what if the merger had a 99b unknown to others?

[sduchesneau]

When it skips, are there no chances of missing forks? How does it know what was in that file in order to skip to the right place, what if the merger had a 99b unknown to others?

it only skips when the merger cannot find enough one-block-files to complete the merge before a mindreader uploads the merged file in its place. A mindreader will only upload that merged file if it is "catching up" (ex: restarted with >100 blocks behind the lib or with these blocks older than 12h...)

so unless you stop the merger for a few minutes, then restart a mindreader from an earlier point in time, let it catch up, then start the merger again, you won't lose any fork. In that particular chain of event, you would just end up losing a block that got forked out, because the correct fork will be in the merged file.

The risk AND impact is minimal and gives you the valid forks.