Block header state, fork database, chainbase, and transaction trace refactors; protocol feature foundations; initial set of new protocol features

[arhag]

Change Description

This PR brings in several changes from the forced-replay branch. They are broken down into the following categories:

• Refactors of existing components
• Protocol feature foundations
• Initial set of new protocol features

All of these changes take advantage of the fact that replay from genesis is forced when upgrading from v1.7.x and earlier to v1.8.x. For example some of the refactors change the fork database structure and chainbase database format. And many changes are made to the indices storing consensus state information in the chainbase state database, which also means changes to what is stored and expected in the snapshot (chain snapshot version has been bumped to version 2) as well as in the state history files and table delta structures sent via its websocket API (no version bumps needed for the state history plugin since it was an alpha product that was expected to have breaking changes).

Refactors of existing components

This PR brings in several changes from prior PRs merged into the forced-replay branch (or the protocol-feature-foundations branch prior to that) that refactored important components nodeos in preparation for the further changes that the protocol feature foundations bring.

The code related to block_header_state and its transitions have been significantly refactored. See PR #6588 for the details of the bulk of the refactoring. However, related PRs #6856 and #6864 fixed a bug resulting from that refactor and added a unit test related to that fixed bug, respectively.

The fork database was refactored in PR #6624. That PR also made significant changes to enable irreversible mode in nodeos. There were some additional bugs remaining with the newly activated irreversible mode that were fixed in PR #6842. Additionally a new Python long-running test was added in PR #6843 to test the additional functions of irreversible mode that unit tests were unable to test.

Also, this PR brings the chainbase submodule up to the latest master which incorporates three PRs merged in the chainbase repository: EOSIO/chainbase#41, EOSIO/chainbase#42, EOSIO/chainbase#43. Of particular importance, is EOSIO/chainbase#42 which uses a new portable header at the start of the shared_memory.bin files used for the chainbase databases to record the environment record with which the database was created with (as well as the dirty flag). This enables a safe and reliable way to check if the current environment matches the environment record stored in the file before opening up the database.

Furthermore, PR #7134 separates out the index to store the actual WebAssembly code from the indices storing account information. It also moves some account-related fields from the account_object to what is now known as the account_metadata_object (formerly the account_sequence_object). PR #7156 fixes a bug introduced in PR #7134.

Finally, PR #7044 restructures transaction traces to use a flatten structure for all the action traces it contains, but with extra metadata information to allow reconstruction of the legacy execution tree as well as the recommended creation tree that provides more information and resolves certain ambiguities that existed with the legacy tree. The push_transaction and push_transactions chain API calls reconstruct the legacy tree before returning the trace back to the caller in an effort to be backwards compatible with existing clients. However, this comes at a cost of additional work that must be done by the server. Updated clients can use the new send_transaction RPC in chain API to get back the new flat transaction trace and avoid the additional computations (and thus slow downs) that push_transaction must do. In additional, PR #7108 adds a new error_code field to the transaction traces.

Protocol features

The foundations for protocol features were implemented in PR #6831. See the PR for details of the changes as well as the some documentation on the new API endpoints that it introduces. That PR also introduces the PREACTIVATE_FEATURE protocol feature which is a critical part of the protocol feature foundations. PR #7167 adds on additional changes to better handle error cases when handling the JSON files specifying the protocol features.

The release that introduces the protocol feature foundations also includes an initial set of protocol features (see #6736 for details). This PR includes implementations for the following set of protocol features that are intended to be included in the initial release:

1. ONLY_LINK_TO_EXISTING_PERMISSION (implemented in PR protocol upgrade activation mechanism (also implements PREACTIVATE_FEATURE and ONLY_LINK_TO_EXISTING_PERMISSION protocol features) #6831 which is the same PR that implements the protocol feature foundations)
2. REPLACE_DEFERRED (Implement REPLACE_DEFERRED protocol feature #6997, add get_account_ram_corrections to producer API #7185)
3. NO_DUPLICATE_DEFERRED_ID (Implement NO_DUPLICATE_DEFERRED_ID protocol feature #7072, Augment protocol_features_tests/no_duplicate_deferred_id_test #7173)
4. FIX_LINKAUTH_RESTRICTION (Implement FIX_LINKAUTH_RESTRICTION protocol feature #7025)
5. DISALLOW_EMPTY_PRODUCER_SCHEDULE (Implement DISALLOW_EMPTY_PRODUCER_SCHEDULE protocol feature #7026)
6. RESTRICT_ACTION_TO_SELF (Implement RESTRICT_ACTION_TO_SELF protocol feature #7088)
7. ONLY_BILL_FIRST_AUTHORIZER (Implement ONLY_BILL_FIRST_AUTHORIZER protocol feature #7089)
8. FORWARD_SETCODE (Implement FORWARD_SETCODE protocol feature #7109)
9. GET_SENDER (Implement GET_SENDER protocol feature #7111)
10. RAM_RESTRICTIONS (Implement RAM_RESTRICTIONS protocol feature #7131)

Other changes

PR #7123 updates the wabt submodule to include changes that may possible improve performance.

PR #7113 makes changes to the structure of files used by the state history plugin. PR #7139 makes adjustments to the state history plugin to avoid outputting table deltas when a modification to the table row does not change any of the fields exposed to state history consumers. PR #7150 augments the transaction_trace structure stored and returned by the state history plugin to include additional data from the original transaction which, in the case of non-delayed input transactions, enables consumers to reconstruct the signed_transaction. PR #7163 adds a limited form of the protocol_state_object table deltas (only including the activated_protocol_features field) to the state history plugin.

PR #7143 updates the bios contract to one supporting the new activate action and updates the tests to work properly with the new bios contract as well as the restrictions of the RESTRICT_ACTION_TO_SELF protocol feature (specifically it updates the bootstrapping procedure of the Python tests to use eosio rather than eosio.token as the token issuer).

PR #7172 updates the behavior of the create_snapshot RPC in the producer_plugin to immediately return the final snapshot path when the node is in the irreversible read mode.

PR #7179 evicts the cached instances of WebAssembly code when the last block in which it existed becomes irreversible.

Consensus Changes

• Consensus Changes

See #6831, #6997, #7072, #7025, #7026, #7088, #7089, #7109, #7111, and #7131.

API Changes

• API Changes

See #6831, #7044, #7113, #7139, #7150, #7163, #7172, and #7185.

Documentation Additions

• Documentation Additions

Documentation changes need to describe the new irreversible mode (see #6624 and #6842 and #6843 for related PRs).

Also see #6831, #7088, #7044, #7108, #7113, #7139, #7150, and #7163.