feat: enable receipt prefetching by default (#7661)

Prefetch receipt meta data (account and access keys) ahead of time.
This recent performance optimization has been disabled by default.

In lab settings, performance improvement is confirmed. Using the
estimator to measure the time it takes to process empty receipts,
on a DB with 2 million accounts, on a local SSD, with enabled shard
caches. The result is as follows.

- sender = receiver:  737us -> 386 us
- sender != receiver: 1014us -> 644us
- overhead per block: 6.9us -> 7.4us

Note that this is with 100 empty receipts in the same block, with all
different accounts. In real traffic it usually does not happen that so
many different accounts are accessed in the same block. But it is
allowed and we must be able process this case in reasonable time.
So even if it might not help in the average case, it makes sense to
activate this feature to speed up the worst-case.

Currently we use 8 IO threads per shard. Repeated experiments with
more threads showed no difference.
Decreasing it to 4 threads is about equal to 8 threads. Going lower is
significantly worse. Thus, overall, 8 threads seems reasonable here.

Canary nodes in testnet and mainnet with the feature enabled show that
the feature also works as expected on real traffic. The memory impact is
minimal, usually less than 40MB of reserved capacity, which is less than
8MB actual memory because 8 threads reserve 4MB each ahead of actually
fetching the data.

CHANGELOG.md

@@ -42,6 +42,11 @@
 * A `[path, data]` JSON RPC query format has been removed.  It has been
   deprecated for over two years and not documented anywhere.  Use proper
   structured queries with `rquest_type` set instead.
+* Enable receipt prefetching by default. This feature makes receipt processing
+  faster by parallelizing IO requests, which has been introduced in
+  [#7590](https://github.com/near/nearcore/pull/7590) and enabled by default
+  with [#7661](https://github.com/near/nearcore/pull/7661).
+  Configurable in `config.json` using `store.enable_receipt_prefetching`.
 
 ## 1.28.0 [2022-07-27]
 

core/store/src/config.rs

@@ -162,7 +162,7 @@ impl Default for StoreConfig {
             block_size: bytesize::ByteSize::kib(16),
 
             trie_cache_capacities: vec![(ShardUId { version: 1, shard_id: 3 }, 45_000_000)],
-            enable_receipt_prefetching: false,
+            enable_receipt_prefetching: true,
             sweat_prefetch_receivers: vec![
                 "token.sweat".to_owned(),
                 "vfinal.token.sweat.testnet".to_owned(),

runtime/runtime-params-estimator/src/testbed.rs

@@ -31,7 +31,17 @@ impl RuntimeTestbed {
             StateDump::from_dir(dump_dir, workdir.path(), in_memory_db);
         // Ensure decent RocksDB SST file layout.
         store.compact().expect("compaction failed");
-        let tries = ShardTries::test(store, 1);
+
+        // Create ShardTries with relevant settings adjusted for estimator.
+        let shard_uids = [ShardUId { shard_id: 0, version: 0 }];
+        let mut trie_config = near_store::TrieConfig::default();
+        trie_config.enable_receipt_prefetching = true;
+        let tries = ShardTries::new(
+            store.clone(),
+            trie_config,
+            &shard_uids,
+            near_store::flat_state::FlatStateFactory::new(store.clone()),
+        );
 
         assert!(roots.len() <= 1, "Parameter estimation works with one shard only.");
         assert!(!roots.is_empty(), "No state roots found.");