[refactor] #2664: Introduce new wsv

[Erigara]

Description

Introduce new in memory storage for iroha.

New approach provide single writer, multiple readers where readers don't block writers and writer doesn't block readers.

Single writer seems reasonable restriction since only single block is processed at the same time.

Instead of single WorldStateView multiple transactions are now used.

Work is based on concread crate.

graph TD;
    WS-->WSV;
    WS-->WSB;
    WSB-->WST;

    WS["State"]
    WSV["StateView"]
    WSB["StateBlock"]
    WST["StateTransaction"]

Loading

• State in-memory iroha storage
• StateView read-only snapshot of state at some point in time
• StateBlock aggregate state changes during block execution (need commit to take effect)
• StateTransaction aggregate state changes during transaction execution (need commit to take effect)

Linked issue

Closes #2664

Benefits

• Faster on large larger state
• Readers and writer are independent

Benchmark results

Iroha tps benchmark was used. In this bechmark only small fraction of state is changed with single transaction (imo it's reasonable to expect that single transaction won't change whole state).

Genesis was generated with the following command:

cargo run --bin kagami genesis --domains 250 --accounts-per-domain 100 --assets-per-domain 100 > configs/peer/genesis.json 

This command creates genesis about 400 Mb in size.

Set the following config for clients/bench/config.json

{
    "peers": 1, // 4 -> 1 To speed up things
    "interval_us_per_tx": 0,
    "max_txs_per_block": 1024,
    "blocks": 15,
    "sample_size": 10,
    "genesis_max_retries": 1200 // Wait for large genesis to be committed
}

Than bench two times before and after changes in wsv.

export WSV_WASM_RUNTIME_CONFIG='{ "FUEL_LIMIT": 18446744073709551615, "MAX_MEMORY": 4294967295 }'
cd client
cargo run --release --example tps-oneshot

tps-oneshot results:

Before update: tps=3.3077273188803833
After update: tps=2339.2057925660447

To measure memory consumption heaptrack was used.

Allocation sizes and amounts:

Before:

After:

All relevant files: link.

TODO

• [ ]: Create relevant issues
• [ ]: Rename no longer relevant variable names, update docs, ...

[coveralls]

Pull Request Test Coverage Report for Build 7097081569

• 1148 of 2221 (51.69%) changed or added relevant lines in 26 files are covered.
• 6436 unchanged lines in 128 files lost coverage.
• Overall coverage decreased (-3.0%) to 56.458%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
core/src/smartcontracts/isi/triggers/mod.rs	0	1	0.0%
core/src/smartcontracts/mod.rs	8	9	88.89%
config/src/iroha.rs	1	3	33.33%
core/src/tx.rs	12	15	80.0%
core/src/smartcontracts/isi/world.rs	0	4	0.0%
core/src/block.rs	38	43	88.37%
telemetry/derive/src/lib.rs	5	10	50.0%
cli/src/lib.rs	0	6	0.0%
core/src/gossiper.rs	0	7	0.0%
core/test_network/src/lib.rs	0	10	0.0%

Files with Coverage Reduction	New Missed Lines	%
config/base/derive/src/view.rs	1	99.37%
config/src/block_sync.rs	1	95.0%
config/src/network.rs	1	93.75%
config/src/torii.rs	1	95.45%
config/src/wasm.rs	1	87.5%
core/src/smartcontracts/isi/block.rs	1	87.5%
config/src/kura.rs	2	79.41%
config/src/lib.rs	2	0.0%
ffi/src/option.rs	2	71.43%
config/src/genesis.rs	3	72.92%

Totals
Change from base Build 5423219773:	-3.0%
Covered Lines:	23311
Relevant Lines:	41289

---

💛 - Coveralls

[SamHSmith]

Good job.

[Arjentix]

New approach provide single reader, multiple readers where readers don't block writers and writer doesn't block readers.

I believe somewhere should be writer instead of reader

[coveralls]

Pull Request Test Coverage Report for Build 8360992827

Details

• 1339 of 2233 (59.96%) changed or added relevant lines in 26 files are covered.
• 4400 unchanged lines in 90 files lost coverage.
• Overall coverage increased (+0.8%) to 57.616%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
core/src/tx.rs	12	15	80.0%
telemetry/derive/src/lib.rs	12	15	80.0%
core/src/smartcontracts/isi/world.rs	0	4	0.0%
cli/src/lib.rs	0	5	0.0%
core/src/block.rs	38	43	88.37%
core/src/smartcontracts/isi/triggers/mod.rs	0	5	0.0%
core/src/gossiper.rs	0	7	0.0%
torii/src/lib.rs	0	7	0.0%
core/test_network/src/lib.rs	0	10	0.0%
core/src/block_sync.rs	0	13	0.0%

Files with Coverage Reduction	New Missed Lines	%
primitives/src/conststr.rs	1	91.14%
ffi/derive/src/convert.rs	1	84.45%
primitives/src/lib.rs	1	0.0%
core/src/sumeragi/network_topology.rs	1	98.78%
telemetry/derive/src/lib.rs	1	64.86%
crypto/src/signature/bls/mod.rs	2	0.0%
config/src/snapshot.rs	3	78.57%
data_model/derive/src/has_origin.rs	3	95.16%
config/src/kura.rs	3	80.0%
primitives/src/addr.rs	3	64.04%

Totals
Change from base Build 7884695009:	0.8%
Covered Lines:	23071
Relevant Lines:	40043

---

💛 - Coveralls

[mversic]

since you found a way to eliminate View vs Snapshot, I have no other complaints here.
I think this change is amazing improvement