feat!: optional versions in transaction inputs

[mrnaveira]

Description

• Refactored transactions so inputs have optional versions. For that I reused the SubstateRequirement struct instead of the SubstateAddress we were using. This change cascaded through all the project.
• Mempool skips execution of transactions that have some inputs without versions. It stores a mock result in database so the transaction is picked up when creating the next HotStuff block. We may want to change this behaviour later (e.g. real execution in mempool, or separated database table for those types of transactions).
• Created a BlockTransactionExecutor used by the consensus when voting on local blocks (I leave remote consensus out of the scope for now). The goal of the executor is to allow the re-execution of transactions taking into account the outputs of previous transactions in the block. The executor will be initialised at the beginning of the block, and will keep a temporary substate memory during the block execution, so each consecutive transaction will read the most recent versions of the inputs from there. After a transaction finishes, the outputs will be used to update the memory.
• After a transaction is executed in consensus, the results are stored in the SQLite database.

Motivation and Context

We want to allow concurrent transactions that affect the same substates. Think for example of a Uniswap-like pool, in which each transaction alters the pool ratio (price impact), in this case transactions must be executed in strict sequential order without conflicts.

The implementation of concurrent transactions is going to spawn across multiple PRs. The goal of this PR is to set the foundation needed for concurrency:

• Allow for clients to send transactions whose inputs don't have specific versions. This is necessary because clients do not know which is the last version of the input they need in case of a high demand component.
• Transaction execution at the HotStuff level instead of at mempool level for transactions without versions. The latest input version is selected for each transaction.

Note that transaction execution and consensus behaviour has not been changed for transactions with specific input versions (the default way of interacting with the network). To trigger the new behaviour the user has to manually craft and send to a VN a transaction that has one or more inputs without version.

Current Limitations and Future Work

• The scope of this PR is concurrent single-shard transactions. Concurrent multi-shard transactions, that would involve more that one committee, are left for future work.
• We may need a separated DB API for substates, with a memory implementation so each block can be processed in parallel without DB conflicts.
• The TariDanBlockTransactionExecutor, which implements the consensus execution of transactions, has some TODO comments for improvements to be made.
• In this PR concurrent transactions do not work (the second onwards fails) due to the behaviour of the pacemaker that we will need to address in future PRs:
  • Blocks seem to be generated very fast, at least in the cucumber scenarios (maybe due to having a single VN network), so it's not possible to force concurrent transactions into the same block. We may need a new network parameter to set up a minimum block time and use it in cucumber tests.
  • Also, there seems to be overlapping of blocks processing. Transactions of block N+x are executed in consensus before transactions in block N are stored into the DB. We will need strict sequential block execution so outputs of one transaction can be used as inputs in another transaction in a future block.

How Has This Been Tested?

New cucumber scenario for concurrent transaction execution. Currently only one transaction without input versions is executed due to the behaviour of the consensus pacemaker (check Current Limitations and Future Work)

What process can a PR reviewer use to test or verify this change?

Run the new cucumber scenario or try to manually send a transaction with no input versions to the VN. I recommend the counter template for testing as it's easy to craft transactions that affect only one substate.

Breaking Changes

• None
• Requires data directory to be deleted
• Network reset. Now transactions inputs contain the substate ID + version, where up until now they were just a substate address (a hash). This change breaks both consensus and clients (UIs, JavaScript, etc.).

[github-actions]

Test Results (CI)

488 tests  +1   488 ✅ +1   1h 46m 35s ⏱️ - 22m 58s
 60 suites +1     0 💤 ±0 
  2 files   ±0     0 ❌ ±0 

Results for commit beb9d7f. ± Comparison against base commit fe428b5.

♻️ This comment has been updated with latest results.

[sdbondi]

ACK

Submitted some transactions and tested and seems to be working.
Nice, looks like a good first step - happy to merge

I suspect the builder interface on the spec can be removed in favour of passing in the executor directly, it seems to be a way to keep the borrow checker happy (immutable self, mutable execute) but this is very minor so no change required at this point.

We'll need to update typescript bindings (this is currently a manual process) by running npm run build in bindings/

[mrnaveira]

We'll need to update typescript bindings (this is currently a manual process) by running npm run build in bindings/

Done