Refactor fork database

[arhag]

Builds on top of PR #6588.

This PR allows irreversible mode to be enabled. It also fixes some bugs that were only triggered during irreversible mode.

This PR also refactors the fork database code to ensure that blocks that become irreversible are immediately appended to the block log, which resolves issues such as #6409.

This refactor changes the format of the portable serialized fork database file that is stored on exit of nodeos (in addition to the existing changes due to #6588). First, the file is saved as "fork_db.dat" rather than "forkdb.dat" to signal that the format has changed in an important way and also to force a replay which is going to be required anyway for other reasons. This new format has a magic number at the start to help distinguish the file as a serialized fork database file, and more importantly it now has a version number to enable future upgrades to the format without necessarily having to abandon support for reading from the old versions going forward.

The new fork database code distinguishes the best head in the fork tree when only considering fully validated blocks it knows about from the best head in the fork tree when considering all blocks it knows about that have at least passed block header validation; this latter type of head is called the pending head in the fork database code.

In modes other than irreversible mode, pending head and the regular head should be identical except for the brief period of time between adding the block to the fork database and applying it in maybe_switch_forks. If the block added to the fork database does not cause any new block to be applied, then the pending head and regular head still remain the same because the pending head would not have changed due to the addition of the block to the fork database (again this is in modes other than irreversible mode).

In irreversible mode, the pending head becomes far more relevant. In that mode, the regular head and pending head are expected to be different. In fact, the regular head block should be the irreversible block from the perspective of the pending head, and so it will be an ancestor of the pending head block.

Two new fields are added to the results returned by get_info in the chain plugin: fork_db_head_block_num and fork_db_head_block_id. These represent the best known head in the fork database tree. The idea is that get_info now provides information for three blocks in the blockchain, where two of the three blocks should always be identical in the get_info response. The three blocks are: the last irreversible block (or last_irreversible_block); the best known block in the fork database (known as the fork_db_head_block); and the block that is tracked by the current consensus state of node (known simply as the head_block). In irreversible mode, the head_block is equivalent to the last_irreversible_block. In all other modes, the head_block is equivalent to the fork_db_head_block.

This PR also adds a unit test(forked_tests/irreversible_mode) which checks that a node in irreversible mode will properly be following the irreversible block even as two other nodes feed it conflicting blocks which cause it to do fork switches.

I have identified further cleanup/refactoring of the code that try to find the appropriate block given a block number, but I am going to hold off on that for the time being so that I can continue making progress with the other features needed for protocol feature foundations.

[heifner]

Would be nice to have a test that failed before that now works for transactions in fork_db. Maybe this is something @brianjohnson5972 or @taokayan could create.

[arhag]

With the changes here and in #6842, nodeos now supports an irreversible read-mode with the ability to easily switch back and forth between irreversible mode and the other read modes.

Switching from speculative mode to irreversible mode

Shut down nodeos and then restart it with the --read-mode=irreversible command line option (the configuration can also be added to the config.ini file).

At startup the nodeos logs should print a warning that looks something like the following:

warn  2019-04-22T21:51:31.318 thread-0  controller.cpp:576            init                 ] database revision (1799) is greater than head block number (1798), attempting to undo pending changes
warn  2019-04-22T21:51:31.320 thread-0  controller.cpp:591            init                 ] read_mode has changed to irreversible: erasing reversible blocks

These warnings will only appear on the first restart that switches from speculative mode (or head or read-only modes) to irreversible mode. Further restarts of nodeos that keep it in irreversible mode will not display these messages.

As the warning states, the reversible blocks database is erased. However, those blocks should still exist in the fork database (which is stored in the fork_db.dat file in the state directory on nodeos exit). This allows nodeos to recover the reversible blocks if it switches back to speculative mode.

While in irreversible mode, the current state always reflects the state as of the end of the last irreversible block. This can be confirmed by comparing the head_block_num with the last_irreversible_block_num in the result of the get_info chain API RPC; the two block numbers should be the same which will likely be less than fork_db_head_block_num.

Switching from irreversible mode to speculative mode

Shut down nodeos while it is configured to be in irreversible mode. Then restart nodeos with the --read-mode=speculative command line option (this is currently the default mode if not read-mode configuration is specified).

At startup the nodeos logs should print a warning that looks something like the following:

warn  2019-04-22T21:59:30.276 thread-0  controller.cpp:657            init                 ] read_mode has changed from irreversible: applying best branch from fork database
warn  2019-04-22T21:59:30.276 thread-0  controller.cpp:663            init                 ] applying branch from fork database ending with block: 00000707f8606d21d07f4342975e677f41369f60514581a81693b3f5c9887ee2

Nodeos will recover the best available branch from the fork database and reapply those blocks. Doing so will add those blocks back into the reversible blocks database. It will also bring the current state back into sync with that of the best head block. This can be confirmed by comparing the head_block_num with the fork_db_head_block_num in the result of the get_info chain API RPC; the two block numbers should be the same which will likely be greater than last_irreversible_block_num.