Parallelize runtime

[MaksymZavershynskyi]

We can easily parallelize execution of smart contracts that touch different accounts. In fact, it might be possible to parallelize execution of smart contracts on the same account using locking. This is a large piece of work and the specific steps will depend on our design of the storage which might change soon. I am leaving out the steps for now.

[vgrichina]

Fine with me. I think it's easier for developer to assume that every async call is "cross-shard" and results in new block vs trying to account for different possible variants.

[MaksymZavershynskyi]

For the record, the hackaton on Dec 21 has demonstrated that the execution of the smart contracts should be parallelized even for the DevNet. When multiple people were submitting contracts to our DevNet we had a congestion which was caused either by the WASM runner or other components of the Runtime.

[ilblackdragon]

To be more specific, the scope of work here:

• Create a pool of runtime workers
  • Concerns: the state should be read from snapshot isolation to prevent other issues with parallel reads? Is there still big lock on the state or we have already removed?
  • Concerns: if we running this pool with actix, what is the performance / parallelism we are getting? Should this be a thread pool directly?
• Organize transactions in the chunk ordered by receiver_id
  • The original grouping can be done when chunk is created (do we do that already?)
• Split by this grouping and feed into a worker pool
  • Receipts and tx that are going to the same receiver_id should be grouped together first
• Join the final state update and other counters (gas etc) from workers

Are there any other caveats? Have @mikhailOK's work for ViewClient already prepared the grounds for lock-free reading in parallel?

[MaksymZavershynskyi]

The caveat is that parallel execution will not account for the worst case scenario -- where all receipts are directed towards the same account. So we need to do a protocol change where we allow at most X gas directed towards the same account. It has the following disadvantages:

• It changes the protocol in a big way, and we might not be accounting for the second-order effects. We already have an experience of thinking some protocol change is seemingly small while missing significant second-order effects. Specifically, state staking -- we thought it is an easy and straightforward change only to find out much later that now we have to attach tokens to many function calls, there is now an attack on the contracts called "state locking". It complicated our contracts, made our DevX worse, and even pushed us to discuss redesigning runtime entirely in the far future;
• Suppose we parallelize the runtime 10x, increase the block gas limit 10x, and say that each recipient can only utilize only 1/10 of the block gas limit. This will push our contract call TPS to 2k. Then imagine one of our partners starts building on us only to discover later that they can only utilize 200 TPS. Sounds like cheating to me. It does not matter how large our marketing TPS will be if many of our partners will not be able to use it. If we want a dishonest marketing TPS number then we can already provide one without parallelizing runtime;
• It will require changes to the mempool iterator, which is a protocol-level change, and we would probably need an additional check in the runtime itself so that it does not need to trust mempool iterator.

The first two bullet points are a no-go to me, so I am closing this issue.