Per-receipt hard limit on storage proof size using upper bound estima…

…tion (#11069)

During receipt execution we record all touched nodes from the pre-state
trie. Those recorded nodes form the storage proof that is sent to
validators, and validators use it to execute the receipts and validate
the results.

In #9378 it's stated that in a
worst case scenario a single receipt can generate hundreds of megabytes
of storage proof. That would cause problems, as it'd cause the
`ChunkStateWitness` to also be hundreds of megabytes in size, and there
would be problems with sending this much data over the network.

Because of that we need to limit the size of the storage proof. We plan
to have two limits:
* per-chunk soft limit - once a chunk has more than X MB of storage
proof we stop processing new receipts, and move the remaining ones to
the delayed receipt queue. This has been implemented in
#10703
* per-receipt hard limit - once a receipt generates more than X MB of
storage proof we fail the receipt, similarly to what happens when a
receipt goes over the allowed gas limit. This one is implemented in this
PR.

Most of the hard-limit code is straightforward - we need to track the
size of recorded storage and fail the receipt if it goes over the limit.
But there is one ugly problem:
#10890. Because of the way
current `TrieUpdate` works we don't record all of the storage proof in
real time. There are some corner cases (deleting one of two children of
a branch) in which some nodes are not recorded until we do `finalize()`
at the end of the chunk. This means that we can't really use
`Trie::recorded_storage_size()` to limit the size, as it isn't fully
accurate. If we do that, a malicious actor could prepare receipts which
seem to have only 1MB of storage proof during execution, but actually
record 10MB during `finalize()`.
There is a long discussion in
#10890 along with some possible
solution ideas, please read that if you need more context.

This PR implements Idea 1 from
#10890.
Instead of using `Trie::recorded_storage_size()` we'll use
`Trie::recorded_storage_size_upper_bound()`, which estimates the upper
bound of recorded storage size by assuming that every trie removal
records additional 2000 bytes:
```rust
    /// Size of the recorded state proof plus some additional size added to cover removals.
    /// An upper-bound estimation of the true recorded size after finalization.
    /// See #10890 and #11000 for details.
    pub fn recorded_storage_size_upper_bound(&self) -> usize {
        // Charge 2000 bytes for every removal
        let removals_size = self.removal_counter.saturating_mul(2000);
        self.recorded_storage_size().saturating_add(removals_size)
    }
```
As long as the upper bound is below the limit we can be sure that the
real recorded size is also below the limit.
It's a rough estimation, which often exaggerates the actual recorded
size (even by 20+ times), but it could be a good-enough/MVP solution for
now. Doing it in a better way would require a lot of refactoring in the
Trie code. We're now [moving
fast](https://near.zulipchat.com/#narrow/stream/407237-core.2Fstateless-validation/topic/Faster.20decision.20making),
so I decided to go with this solution for now.

The upper bound calculation has been added in a previous PR along with
the metrics to see if using such a rough estimation is viable:
#11000

I set up a mainnet node with shadow validation to gather some data about
the size distribution with mainnet traffic: [Metrics
link](https://nearinc.grafana.net/d/edbl9ztm5h1q8b/stateless-validation?orgId=1&var-chain_id=mainnet&var-shard_id=All&var-node_id=ci-b20a9aef-mainnet-rpc-europe-west4-01-84346caf&from=1713225600000&to=1713272400000)

![image](https://github.com/near/nearcore/assets/149345204/dc3daa88-5f59-4ae5-aa9e-ab2802f034b8)

![image](https://github.com/near/nearcore/assets/149345204/90602443-7a0f-4503-9bce-8fbce352c0ba)

The metrics show that:
* For all receipts both the recorded size and the upper bound estimate
are below 2MB
* Overwhelming majority of receipts generate < 50KB of storage proof
* For all chunks the upper bound estimate is below 6MB
* For 99% of chunks the upper bound estimate is below 3MB

Based on this I believe that we can:
* Set the hard per-receipt limit to 4MB. All receipts were below 2MB,
but it's good to have a bit of a safety margin here. This is a hard
limit, so it might break existing contracts if they turn out to generate
more storage proof than the limit.
* Set the soft per-chunk limit to 3MB. 99% of chunks will not be
affected by this limit. For the 1% that hit the limit they'll execute
fewer receipts, with the rest of the receipts put into the delayed
receipt queue. This slightly lowers throughput of a single chunk, but
it's not a big slowdown, by ~1%.

Having a 4MB per-receipt hard limit and a 3MB per-chunk soft limit would
give us a hard guarantee that for all chunks the total storage proof
size is below 7MB.

It's worth noting that gas usage already limits the storage proof size
quite effectively. For 98% of chunks the storage proof size is already
below 2MB, so the limit isn't really needed for typical mainnet traffic.
The limit matters mostly for stopping malicious actors that'd try to DoS
the network by generating large storage proofs.

Fixes: #11019

core/parameters/res/runtime_configs/85.yaml

@@ -0,0 +1,2 @@
+storage_proof_size_receipt_limit: {old: 999_999_999_999_999, new: 4_000_000}
+storage_proof_size_soft_limit: {old: 16_000_000, new: 3_000_000}

core/parameters/res/runtime_configs/parameters.snap

@@ -5,6 +5,7 @@ description: THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
 burnt_gas_reward                        3 / 10
 pessimistic_gas_price_inflation         103 / 100
 storage_proof_size_soft_limit            999_999_999_999_999
+storage_proof_size_receipt_limit         999_999_999_999_999
 min_allowed_top_level_account_length                      65
 registrar_account_id                    registrar
 storage_amount_per_byte                 10000000000000000000

core/parameters/res/runtime_configs/parameters.yaml

@@ -14,6 +14,7 @@ pessimistic_gas_price_inflation: {
 
 # Stateless validation config
 storage_proof_size_soft_limit: 999_999_999_999_999
+storage_proof_size_receipt_limit: 999_999_999_999_999
 
 # Account creation config
 min_allowed_top_level_account_length: 32

core/parameters/res/runtime_configs/parameters_testnet.yaml

@@ -10,6 +10,7 @@ pessimistic_gas_price_inflation: {
 
 # Stateless validation config
 storage_proof_size_soft_limit: 999_999_999_999_999
+storage_proof_size_receipt_limit: 999_999_999_999_999
 
 # Account creation config
 min_allowed_top_level_account_length: 0

core/parameters/src/config_store.rs

@@ -39,6 +39,7 @@ static CONFIG_DIFFS: &[(ProtocolVersion, &str)] = &[
     (64, include_config!("64.yaml")),
     (66, include_config!("66.yaml")),
     (83, include_config!("83.yaml")),
+    (85, include_config!("85.yaml")),
     (129, include_config!("129.yaml")),
     // Introduce ETH-implicit accounts.
     (138, include_config!("138.yaml")),

core/parameters/src/parameter.rs

@@ -22,6 +22,8 @@ pub enum Parameter {
 
     // Stateless validation config
     StorageProofSizeSoftLimit,
+    // Hard per-receipt limit of recorded trie storage proof
+    StorageProofSizeReceiptLimit,
 
     // Account creation config
     MinAllowedTopLevelAccountLength,
@@ -234,6 +236,7 @@ impl Parameter {
             Parameter::AccountIdValidityRulesVersion,
             Parameter::YieldTimeoutLengthInBlocks,
             Parameter::MaxYieldPayloadSize,
+            Parameter::StorageProofSizeReceiptLimit,
         ]
         .iter()
     }

core/parameters/src/snapshots/near_parameters__config_store__tests__0.json.snap

@@ -214,7 +214,8 @@ expression: config_view
       "wasmer2_stack_limit": 102400,
       "account_id_validity_rules_version": 0,
       "yield_timeout_length_in_blocks": 200,
-      "max_yield_payload_size": 1024
+      "max_yield_payload_size": 1024,
+      "storage_proof_size_receipt_limit": 999999999999999
     }
   },
   "account_creation_config": {

core/parameters/src/snapshots/near_parameters__config_store__tests__129.json.snap

@@ -103,7 +103,7 @@ expression: config_view
       103,
       100
     ],
-    "storage_proof_size_soft_limit": 16000000
+    "storage_proof_size_soft_limit": 3000000
   },
   "wasm_config": {
     "ext_costs": {
@@ -216,7 +216,8 @@ expression: config_view
       "max_locals_per_contract": 1000000,
       "account_id_validity_rules_version": 1,
       "yield_timeout_length_in_blocks": 200,
-      "max_yield_payload_size": 1024
+      "max_yield_payload_size": 1024,
+      "storage_proof_size_receipt_limit": 4000000
     }
   },
   "account_creation_config": {

core/parameters/src/snapshots/near_parameters__config_store__tests__138.json.snap

@@ -103,7 +103,7 @@ expression: config_view
       103,
       100
     ],
-    "storage_proof_size_soft_limit": 16000000
+    "storage_proof_size_soft_limit": 3000000
   },
   "wasm_config": {
     "ext_costs": {
@@ -216,7 +216,8 @@ expression: config_view
       "max_locals_per_contract": 1000000,
       "account_id_validity_rules_version": 1,
       "yield_timeout_length_in_blocks": 200,
-      "max_yield_payload_size": 1024
+      "max_yield_payload_size": 1024,
+      "storage_proof_size_receipt_limit": 4000000
     }
   },
   "account_creation_config": {

core/parameters/src/snapshots/near_parameters__config_store__tests__139.json.snap

@@ -103,7 +103,7 @@ expression: config_view
       103,
       100
     ],
-    "storage_proof_size_soft_limit": 16000000
+    "storage_proof_size_soft_limit": 3000000
   },
   "wasm_config": {
     "ext_costs": {
@@ -216,7 +216,8 @@ expression: config_view
       "max_locals_per_contract": 1000000,
       "account_id_validity_rules_version": 1,
       "yield_timeout_length_in_blocks": 200,
-      "max_yield_payload_size": 1024
+      "max_yield_payload_size": 1024,
+      "storage_proof_size_receipt_limit": 4000000
     }
   },
   "account_creation_config": {

core/parameters/src/snapshots/near_parameters__config_store__tests__35.json.snap

@@ -214,7 +214,8 @@ expression: config_view
       "wasmer2_stack_limit": 102400,
       "account_id_validity_rules_version": 0,
       "yield_timeout_length_in_blocks": 200,
-      "max_yield_payload_size": 1024
+      "max_yield_payload_size": 1024,
+      "storage_proof_size_receipt_limit": 999999999999999
     }
   },
   "account_creation_config": {

core/parameters/src/snapshots/near_parameters__config_store__tests__42.json.snap

@@ -214,7 +214,8 @@ expression: config_view
       "wasmer2_stack_limit": 102400,
       "account_id_validity_rules_version": 0,
       "yield_timeout_length_in_blocks": 200,
-      "max_yield_payload_size": 1024
+      "max_yield_payload_size": 1024,
+      "storage_proof_size_receipt_limit": 999999999999999
     }
   },
   "account_creation_config": {

core/parameters/src/snapshots/near_parameters__config_store__tests__46.json.snap

@@ -214,7 +214,8 @@ expression: config_view
       "wasmer2_stack_limit": 102400,
       "account_id_validity_rules_version": 0,
       "yield_timeout_length_in_blocks": 200,
-      "max_yield_payload_size": 1024
+      "max_yield_payload_size": 1024,
+      "storage_proof_size_receipt_limit": 999999999999999
     }
   },
   "account_creation_config": {

core/parameters/src/snapshots/near_parameters__config_store__tests__48.json.snap

@@ -214,7 +214,8 @@ expression: config_view
       "wasmer2_stack_limit": 102400,
       "account_id_validity_rules_version": 0,
       "yield_timeout_length_in_blocks": 200,
-      "max_yield_payload_size": 1024
+      "max_yield_payload_size": 1024,
+      "storage_proof_size_receipt_limit": 999999999999999
     }
   },
   "account_creation_config": {

core/parameters/src/snapshots/near_parameters__config_store__tests__49.json.snap

@@ -215,7 +215,8 @@ expression: config_view
       "wasmer2_stack_limit": 102400,
       "account_id_validity_rules_version": 0,
       "yield_timeout_length_in_blocks": 200,
-      "max_yield_payload_size": 1024
+      "max_yield_payload_size": 1024,
+      "storage_proof_size_receipt_limit": 999999999999999
     }
   },
   "account_creation_config": {

core/parameters/src/snapshots/near_parameters__config_store__tests__50.json.snap

@@ -215,7 +215,8 @@ expression: config_view
       "wasmer2_stack_limit": 102400,
       "account_id_validity_rules_version": 0,
       "yield_timeout_length_in_blocks": 200,
-      "max_yield_payload_size": 1024
+      "max_yield_payload_size": 1024,
+      "storage_proof_size_receipt_limit": 999999999999999
     }
   },
   "account_creation_config": {

core/parameters/src/snapshots/near_parameters__config_store__tests__52.json.snap

@@ -215,7 +215,8 @@ expression: config_view
       "wasmer2_stack_limit": 102400,
       "account_id_validity_rules_version": 0,
       "yield_timeout_length_in_blocks": 200,
-      "max_yield_payload_size": 1024
+      "max_yield_payload_size": 1024,
+      "storage_proof_size_receipt_limit": 999999999999999
     }
   },
   "account_creation_config": {

core/parameters/src/snapshots/near_parameters__config_store__tests__53.json.snap

@@ -216,7 +216,8 @@ expression: config_view
       "max_locals_per_contract": 1000000,
       "account_id_validity_rules_version": 0,
       "yield_timeout_length_in_blocks": 200,
-      "max_yield_payload_size": 1024
+      "max_yield_payload_size": 1024,
+      "storage_proof_size_receipt_limit": 999999999999999
     }
   },
   "account_creation_config": {

core/parameters/src/snapshots/near_parameters__config_store__tests__55.json.snap

@@ -216,7 +216,8 @@ expression: config_view
       "max_locals_per_contract": 1000000,
       "account_id_validity_rules_version": 0,
       "yield_timeout_length_in_blocks": 200,
-      "max_yield_payload_size": 1024
+      "max_yield_payload_size": 1024,
+      "storage_proof_size_receipt_limit": 999999999999999
     }
   },
   "account_creation_config": {

core/parameters/src/snapshots/near_parameters__config_store__tests__57.json.snap

@@ -216,7 +216,8 @@ expression: config_view
       "max_locals_per_contract": 1000000,
       "account_id_validity_rules_version": 1,
       "yield_timeout_length_in_blocks": 200,
-      "max_yield_payload_size": 1024
+      "max_yield_payload_size": 1024,
+      "storage_proof_size_receipt_limit": 999999999999999
     }
   },
   "account_creation_config": {

core/parameters/src/snapshots/near_parameters__config_store__tests__59.json.snap

@@ -216,7 +216,8 @@ expression: config_view
       "max_locals_per_contract": 1000000,
       "account_id_validity_rules_version": 1,
       "yield_timeout_length_in_blocks": 200,
-      "max_yield_payload_size": 1024
+      "max_yield_payload_size": 1024,
+      "storage_proof_size_receipt_limit": 999999999999999
     }
   },
   "account_creation_config": {

core/parameters/src/snapshots/near_parameters__config_store__tests__61.json.snap

@@ -216,7 +216,8 @@ expression: config_view
       "max_locals_per_contract": 1000000,
       "account_id_validity_rules_version": 1,
       "yield_timeout_length_in_blocks": 200,
-      "max_yield_payload_size": 1024
+      "max_yield_payload_size": 1024,
+      "storage_proof_size_receipt_limit": 999999999999999
     }
   },
   "account_creation_config": {

core/parameters/src/snapshots/near_parameters__config_store__tests__62.json.snap

@@ -216,7 +216,8 @@ expression: config_view
       "max_locals_per_contract": 1000000,
       "account_id_validity_rules_version": 1,
       "yield_timeout_length_in_blocks": 200,
-      "max_yield_payload_size": 1024
+      "max_yield_payload_size": 1024,
+      "storage_proof_size_receipt_limit": 999999999999999
     }
   },
   "account_creation_config": {

core/parameters/src/snapshots/near_parameters__config_store__tests__63.json.snap

@@ -216,7 +216,8 @@ expression: config_view
       "max_locals_per_contract": 1000000,
       "account_id_validity_rules_version": 1,
       "yield_timeout_length_in_blocks": 200,
-      "max_yield_payload_size": 1024
+      "max_yield_payload_size": 1024,
+      "storage_proof_size_receipt_limit": 999999999999999
     }
   },
   "account_creation_config": {

core/parameters/src/snapshots/near_parameters__config_store__tests__64.json.snap

@@ -216,7 +216,8 @@ expression: config_view
       "max_locals_per_contract": 1000000,
       "account_id_validity_rules_version": 1,
       "yield_timeout_length_in_blocks": 200,
-      "max_yield_payload_size": 1024
+      "max_yield_payload_size": 1024,
+      "storage_proof_size_receipt_limit": 999999999999999
     }
   },
   "account_creation_config": {

core/parameters/src/snapshots/near_parameters__config_store__tests__66.json.snap

@@ -216,7 +216,8 @@ expression: config_view
       "max_locals_per_contract": 1000000,
       "account_id_validity_rules_version": 1,
       "yield_timeout_length_in_blocks": 200,
-      "max_yield_payload_size": 1024
+      "max_yield_payload_size": 1024,
+      "storage_proof_size_receipt_limit": 999999999999999
     }
   },
   "account_creation_config": {

core/parameters/src/snapshots/near_parameters__config_store__tests__83.json.snap

@@ -216,7 +216,8 @@ expression: config_view
       "max_locals_per_contract": 1000000,
       "account_id_validity_rules_version": 1,
       "yield_timeout_length_in_blocks": 200,
-      "max_yield_payload_size": 1024
+      "max_yield_payload_size": 1024,
+      "storage_proof_size_receipt_limit": 999999999999999
     }
   },
   "account_creation_config": {