Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(stateless_validation): Limit size of outgoing receipts to keep size of source_receipt_proofs under control #11492

Merged
merged 21 commits into from
Jun 7, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
86d5b84
Add new runtime parameters
jancionear Jun 5, 2024
747078d
Rename CongestionControl::outgoing_limit to outgoing_gas_limit
jancionear Jun 5, 2024
6287f4e
Limit size of outgoing receipts to keep the size of source_receipt_pr…
jancionear Jun 5, 2024
7b94e00
Fix meta_tx_create_near_implicit_account_fails test
jancionear Jun 5, 2024
3cbb76a
Fix test_protocol_upgrade_under_congestion
jancionear Jun 5, 2024
3ce4357
fix json.snap file using cargo insta review
jancionear Jun 5, 2024
d688614
Merge branch 'master' into outsized
jancionear Jun 7, 2024
1ed28b1
Add comments on the Parameter enum
jancionear Jun 7, 2024
dcb9ccf
fix receieves typo
jancionear Jun 7, 2024
f749388
Fn -> FnOnce
jancionear Jun 7, 2024
c17e0f0
Use u64 instead of usize
jancionear Jun 7, 2024
cc6ff44
Use allowed_shard instead of special_shard
jancionear Jun 7, 2024
c17d2cb
Disable allowed_shard validation, it's buggy and Wac is working on a fix
jancionear Jun 7, 2024
8f8a73b
Add receipt size limit
jancionear Jun 7, 2024
33ab33c
Move the changes to protocol version 87, don't add a new one
jancionear Jun 7, 2024
72e5724
Use #[ignore] instead of commenting out test
jancionear Jun 7, 2024
cc019c8
Add comment saying that the limit is aggregate, not per-receipt
jancionear Jun 7, 2024
1736fc6
Add one more TODO about allowed_shard validation
jancionear Jun 7, 2024
409e40e
Refactor setting gas limit to avoid mutability
jancionear Jun 7, 2024
230efcf
Update rpc_errors_schema.json to fix CI
jancionear Jun 7, 2024
c0ee219
Move outgoing size limits to CongestionControlConfig
jancionear Jun 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions chain/client/src/tests/process_blocks.rs
Original file line number Diff line number Diff line change
Expand Up @@ -255,6 +255,8 @@ fn test_bad_congestion_info_corrupt_buffered_receipts_bytes() {
test_bad_congestion_info_impl(BadCongestionInfoMode::CorruptBufferedReceiptsBytes);
}

// TODO(congestion_control) validate allowed shard
#[ignore]
#[test]
fn test_bad_congestion_info_corrupt_allowed_shard() {
test_bad_congestion_info_impl(BadCongestionInfoMode::CorruptAllowedShard);
Expand Down
11 changes: 10 additions & 1 deletion chain/jsonrpc/res/rpc_errors_schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -735,6 +735,14 @@
"method_name": ""
}
},
"ReceiptSizeExceeded": {
"name": "ReceiptSizeExceeded",
"subtypes": [],
"props": {
"limit": "",
"size": ""
}
},
"ReceiptValidationError": {
"name": "ReceiptValidationError",
"subtypes": [
Expand All @@ -744,7 +752,8 @@
"InvalidDataReceiverId",
"ReturnedValueLengthExceeded",
"NumberInputDataDependenciesExceeded",
"ActionsValidation"
"ActionsValidation",
"ReceiptSizeExceeded"
],
"props": {}
},
Expand Down
6 changes: 6 additions & 0 deletions core/parameters/res/runtime_configs/87.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
# State Witness

max_transaction_size: {old: 4_194_304, new: 1_572_864}
max_receipt_size: {old: 999_999_999_999_999, new: 4_194_304}
Comment on lines 4 to +5
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to use the same value for transaction and receipt limits? It seems a bit more intuitive. Do you have any data on whether there were any receipts above that limit in the history?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be possible to lower the receipt limit further, but it would break some receipts, there are some 3.9MB ones. It would require some more thinking about, for now 4MiB seems like a safe and good enough choice, we can lower it in a follow-up PR.
https://near.zulipchat.com/#narrow/stream/295306-contract-runtime/topic/How.20large.20can.20internal.20receipts.20get.3F/near/443309601

combined_transactions_size_limit: {old: 999_999_999_999_999, new: 2_097_152}
new_transactions_validation_state_size_soft_limit: {old: 999_999_999_999_999, new: 572_864}

Expand Down Expand Up @@ -70,3 +71,8 @@ reject_tx_congestion_threshold: {
new : { numerator: 25, denominator: 100 }
}

# 100 kiB
outgoing_receipts_usual_size_limit: {old: 999_999_999_999_999, new: 102_400}

# 4.5 MiB
outgoing_receipts_big_size_limit: {old: 999_999_999_999_999, new: 4_718_592}
3 changes: 3 additions & 0 deletions core/parameters/res/runtime_configs/parameters.snap
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ main_storage_proof_size_soft_limit 999_999_999_999_999
per_receipt_storage_proof_size_limit 999_999_999_999_999
new_transactions_validation_state_size_soft_limit 999_999_999_999_999
combined_transactions_size_limit 999_999_999_999_999
outgoing_receipts_usual_size_limit 999_999_999_999_999
outgoing_receipts_big_size_limit 999_999_999_999_999
min_allowed_top_level_account_length 65
registrar_account_id registrar
storage_amount_per_byte 10000000000000000000
Expand Down Expand Up @@ -163,6 +165,7 @@ max_arguments_length 4_194_304
max_length_returned_data 4_194_304
max_contract_size 4_194_304
max_transaction_size 4_194_304
max_receipt_size 999_999_999_999_999
max_length_storage_key 2_048
max_length_storage_value 4_194_304
max_promises_per_function_call_action 1_024
Expand Down
3 changes: 3 additions & 0 deletions core/parameters/res/runtime_configs/parameters.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ main_storage_proof_size_soft_limit: 999_999_999_999_999
per_receipt_storage_proof_size_limit: 999_999_999_999_999
combined_transactions_size_limit: 999_999_999_999_999
new_transactions_validation_state_size_soft_limit: 999_999_999_999_999
outgoing_receipts_usual_size_limit: 999_999_999_999_999
outgoing_receipts_big_size_limit: 999_999_999_999_999

# Account creation config
min_allowed_top_level_account_length: 32
Expand Down Expand Up @@ -201,6 +203,7 @@ max_arguments_length: 4_194_304
max_length_returned_data: 4_194_304
max_contract_size: 4_194_304
max_transaction_size: 4_194_304
max_receipt_size: 999_999_999_999_999
max_length_storage_key: 4_194_304
max_length_storage_value: 4_194_304
max_promises_per_function_call_action: 1_024
Expand Down
3 changes: 3 additions & 0 deletions core/parameters/res/runtime_configs/parameters_testnet.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ main_storage_proof_size_soft_limit: 999_999_999_999_999
per_receipt_storage_proof_size_limit: 999_999_999_999_999
combined_transactions_size_limit: 999_999_999_999_999
new_transactions_validation_state_size_soft_limit: 999_999_999_999_999
outgoing_receipts_usual_size_limit: 999_999_999_999_999
outgoing_receipts_big_size_limit: 999_999_999_999_999

# Account creation config
min_allowed_top_level_account_length: 0
Expand Down Expand Up @@ -198,6 +200,7 @@ max_arguments_length: 4_194_304
max_length_returned_data: 4_194_304
max_contract_size: 4_194_304
max_transaction_size: 4_194_304
max_receipt_size: 999_999_999_999_999
max_length_storage_key: 4_194_304
max_length_storage_value: 4_194_304
max_promises_per_function_call_action: 1_024
Expand Down
12 changes: 12 additions & 0 deletions core/parameters/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,16 @@ pub struct CongestionControlConfig {
/// How much congestion a shard can tolerate before it stops all shards from
/// accepting new transactions with the receiver set to the congested shard.
pub reject_tx_congestion_threshold: f64,

/// The standard size limit for outgoing receipts aimed at a single shard.
/// This limit is pretty small to keep the size of source_receipt_proofs under control.
/// It limits the total sum of outgoing receipts, not individual receipts.
pub outgoing_receipts_usual_size_limit: u64,

/// Large size limit for outgoing receipts to a shard, used when it's safe
/// to send a lot of receipts without making the state witness too large.
/// It limits the total sum of outgoing receipts, not individual receipts.
pub outgoing_receipts_big_size_limit: u64,
}

// The Eq cannot be automatically derived for this class because it contains a
Expand All @@ -204,6 +214,8 @@ impl CongestionControlConfig {
max_tx_gas: max_value,
min_tx_gas: max_value,
reject_tx_congestion_threshold: 2.0,
outgoing_receipts_usual_size_limit: max_value,
outgoing_receipts_big_size_limit: max_value,
}
}
}
Expand Down
10 changes: 10 additions & 0 deletions core/parameters/src/parameter.rs
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,14 @@ pub enum Parameter {
/// A witness contains transactions from both the previous chunk and the current one.
/// This parameter limits the sum of sizes of transactions from both of those chunks.
CombinedTransactionsSizeLimit,
/// The standard size limit for outgoing receipts aimed at a single shard.
/// This limit is pretty small to keep the size of source_receipt_proofs under control.
/// It limits the total sum of outgoing receipts, not individual receipts.
OutgoingReceiptsUsualSizeLimit,
/// Large size limit for outgoing receipts to a shard, used when it's safe
/// to send a lot of receipts without making the state witness too large.
/// It limits the total sum of outgoing receipts, not individual receipts.
OutgoingReceiptsBigSizeLimit,

// Account creation config
MinAllowedTopLevelAccountLength,
Expand Down Expand Up @@ -153,6 +161,7 @@ pub enum Parameter {
MaxLengthReturnedData,
MaxContractSize,
MaxTransactionSize,
MaxReceiptSize,
MaxLengthStorageKey,
MaxLengthStorageValue,
MaxPromisesPerFunctionCallAction,
Expand Down Expand Up @@ -247,6 +256,7 @@ impl Parameter {
Parameter::MaxLengthReturnedData,
Parameter::MaxContractSize,
Parameter::MaxTransactionSize,
Parameter::MaxReceiptSize,
Parameter::MaxLengthStorageKey,
Parameter::MaxLengthStorageValue,
Parameter::MaxPromisesPerFunctionCallAction,
Expand Down
3 changes: 3 additions & 0 deletions core/parameters/src/parameter_table.rs
Original file line number Diff line number Diff line change
Expand Up @@ -363,6 +363,9 @@ fn get_congestion_control_config(
let rational: Rational32 = params.get(Parameter::RejectTxCongestionThreshold)?;
*rational.numer() as f64 / *rational.denom() as f64
},
outgoing_receipts_usual_size_limit: params
.get(Parameter::OutgoingReceiptsUsualSizeLimit)?,
outgoing_receipts_big_size_limit: params.get(Parameter::OutgoingReceiptsBigSizeLimit)?,
};
Ok(congestion_control_config)
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,7 @@ expression: config_view
"max_length_returned_data": 4194304,
"max_contract_size": 4194304,
"max_transaction_size": 4194304,
"max_receipt_size": 999999999999999,
"max_length_storage_key": 4194304,
"max_length_storage_value": 4194304,
"max_promises_per_function_call_action": 1024,
Expand All @@ -231,7 +232,9 @@ expression: config_view
"allowed_shard_outgoing_gas": 9223372036854775807,
"max_tx_gas": 9223372036854775807,
"min_tx_gas": 9223372036854775807,
"reject_tx_congestion_threshold": 1.0
"reject_tx_congestion_threshold": 1.0,
"outgoing_receipts_usual_size_limit": 999999999999999,
"outgoing_receipts_big_size_limit": 999999999999999
},
"witness_config": {
"main_storage_proof_size_soft_limit": 999999999999999,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,7 @@ expression: config_view
"max_length_returned_data": 4194304,
"max_contract_size": 4194304,
"max_transaction_size": 1572864,
"max_receipt_size": 4194304,
"max_length_storage_key": 2048,
"max_length_storage_value": 4194304,
"max_promises_per_function_call_action": 1024,
Expand Down Expand Up @@ -233,7 +234,9 @@ expression: config_view
"allowed_shard_outgoing_gas": 1000000000000000,
"max_tx_gas": 500000000000000,
"min_tx_gas": 20000000000000,
"reject_tx_congestion_threshold": 0.25
"reject_tx_congestion_threshold": 0.25,
"outgoing_receipts_usual_size_limit": 102400,
"outgoing_receipts_big_size_limit": 4718592
},
"witness_config": {
"main_storage_proof_size_soft_limit": 3000000,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,7 @@ expression: config_view
"max_length_returned_data": 4194304,
"max_contract_size": 4194304,
"max_transaction_size": 1572864,
"max_receipt_size": 4194304,
"max_length_storage_key": 2048,
"max_length_storage_value": 4194304,
"max_promises_per_function_call_action": 1024,
Expand Down Expand Up @@ -233,7 +234,9 @@ expression: config_view
"allowed_shard_outgoing_gas": 1000000000000000,
"max_tx_gas": 500000000000000,
"min_tx_gas": 20000000000000,
"reject_tx_congestion_threshold": 0.25
"reject_tx_congestion_threshold": 0.25,
"outgoing_receipts_usual_size_limit": 102400,
"outgoing_receipts_big_size_limit": 4718592
},
"witness_config": {
"main_storage_proof_size_soft_limit": 3000000,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,7 @@ expression: config_view
"max_length_returned_data": 4194304,
"max_contract_size": 4194304,
"max_transaction_size": 4194304,
"max_receipt_size": 999999999999999,
"max_length_storage_key": 4194304,
"max_length_storage_value": 4194304,
"max_promises_per_function_call_action": 1024,
Expand All @@ -231,7 +232,9 @@ expression: config_view
"allowed_shard_outgoing_gas": 9223372036854775807,
"max_tx_gas": 9223372036854775807,
"min_tx_gas": 9223372036854775807,
"reject_tx_congestion_threshold": 1.0
"reject_tx_congestion_threshold": 1.0,
"outgoing_receipts_usual_size_limit": 999999999999999,
"outgoing_receipts_big_size_limit": 999999999999999
},
"witness_config": {
"main_storage_proof_size_soft_limit": 999999999999999,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,7 @@ expression: config_view
"max_length_returned_data": 4194304,
"max_contract_size": 4194304,
"max_transaction_size": 4194304,
"max_receipt_size": 999999999999999,
"max_length_storage_key": 4194304,
"max_length_storage_value": 4194304,
"max_promises_per_function_call_action": 1024,
Expand All @@ -231,7 +232,9 @@ expression: config_view
"allowed_shard_outgoing_gas": 9223372036854775807,
"max_tx_gas": 9223372036854775807,
"min_tx_gas": 9223372036854775807,
"reject_tx_congestion_threshold": 1.0
"reject_tx_congestion_threshold": 1.0,
"outgoing_receipts_usual_size_limit": 999999999999999,
"outgoing_receipts_big_size_limit": 999999999999999
},
"witness_config": {
"main_storage_proof_size_soft_limit": 999999999999999,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,7 @@ expression: config_view
"max_length_returned_data": 4194304,
"max_contract_size": 4194304,
"max_transaction_size": 4194304,
"max_receipt_size": 999999999999999,
"max_length_storage_key": 4194304,
"max_length_storage_value": 4194304,
"max_promises_per_function_call_action": 1024,
Expand All @@ -231,7 +232,9 @@ expression: config_view
"allowed_shard_outgoing_gas": 9223372036854775807,
"max_tx_gas": 9223372036854775807,
"min_tx_gas": 9223372036854775807,
"reject_tx_congestion_threshold": 1.0
"reject_tx_congestion_threshold": 1.0,
"outgoing_receipts_usual_size_limit": 999999999999999,
"outgoing_receipts_big_size_limit": 999999999999999
},
"witness_config": {
"main_storage_proof_size_soft_limit": 999999999999999,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,7 @@ expression: config_view
"max_length_returned_data": 4194304,
"max_contract_size": 4194304,
"max_transaction_size": 4194304,
"max_receipt_size": 999999999999999,
"max_length_storage_key": 4194304,
"max_length_storage_value": 4194304,
"max_promises_per_function_call_action": 1024,
Expand All @@ -231,7 +232,9 @@ expression: config_view
"allowed_shard_outgoing_gas": 9223372036854775807,
"max_tx_gas": 9223372036854775807,
"min_tx_gas": 9223372036854775807,
"reject_tx_congestion_threshold": 1.0
"reject_tx_congestion_threshold": 1.0,
"outgoing_receipts_usual_size_limit": 999999999999999,
"outgoing_receipts_big_size_limit": 999999999999999
},
"witness_config": {
"main_storage_proof_size_soft_limit": 999999999999999,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,7 @@ expression: config_view
"max_length_returned_data": 4194304,
"max_contract_size": 4194304,
"max_transaction_size": 4194304,
"max_receipt_size": 999999999999999,
"max_length_storage_key": 4194304,
"max_length_storage_value": 4194304,
"max_promises_per_function_call_action": 1024,
Expand All @@ -232,7 +233,9 @@ expression: config_view
"allowed_shard_outgoing_gas": 9223372036854775807,
"max_tx_gas": 9223372036854775807,
"min_tx_gas": 9223372036854775807,
"reject_tx_congestion_threshold": 1.0
"reject_tx_congestion_threshold": 1.0,
"outgoing_receipts_usual_size_limit": 999999999999999,
"outgoing_receipts_big_size_limit": 999999999999999
},
"witness_config": {
"main_storage_proof_size_soft_limit": 999999999999999,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,7 @@ expression: config_view
"max_length_returned_data": 4194304,
"max_contract_size": 4194304,
"max_transaction_size": 4194304,
"max_receipt_size": 999999999999999,
"max_length_storage_key": 4194304,
"max_length_storage_value": 4194304,
"max_promises_per_function_call_action": 1024,
Expand All @@ -232,7 +233,9 @@ expression: config_view
"allowed_shard_outgoing_gas": 9223372036854775807,
"max_tx_gas": 9223372036854775807,
"min_tx_gas": 9223372036854775807,
"reject_tx_congestion_threshold": 1.0
"reject_tx_congestion_threshold": 1.0,
"outgoing_receipts_usual_size_limit": 999999999999999,
"outgoing_receipts_big_size_limit": 999999999999999
},
"witness_config": {
"main_storage_proof_size_soft_limit": 999999999999999,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,7 @@ expression: config_view
"max_length_returned_data": 4194304,
"max_contract_size": 4194304,
"max_transaction_size": 4194304,
"max_receipt_size": 999999999999999,
"max_length_storage_key": 4194304,
"max_length_storage_value": 4194304,
"max_promises_per_function_call_action": 1024,
Expand All @@ -232,7 +233,9 @@ expression: config_view
"allowed_shard_outgoing_gas": 9223372036854775807,
"max_tx_gas": 9223372036854775807,
"min_tx_gas": 9223372036854775807,
"reject_tx_congestion_threshold": 1.0
"reject_tx_congestion_threshold": 1.0,
"outgoing_receipts_usual_size_limit": 999999999999999,
"outgoing_receipts_big_size_limit": 999999999999999
},
"witness_config": {
"main_storage_proof_size_soft_limit": 999999999999999,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,7 @@ expression: config_view
"max_length_returned_data": 4194304,
"max_contract_size": 4194304,
"max_transaction_size": 4194304,
"max_receipt_size": 999999999999999,
"max_length_storage_key": 2048,
"max_length_storage_value": 4194304,
"max_promises_per_function_call_action": 1024,
Expand Down Expand Up @@ -233,7 +234,9 @@ expression: config_view
"allowed_shard_outgoing_gas": 9223372036854775807,
"max_tx_gas": 9223372036854775807,
"min_tx_gas": 9223372036854775807,
"reject_tx_congestion_threshold": 1.0
"reject_tx_congestion_threshold": 1.0,
"outgoing_receipts_usual_size_limit": 999999999999999,
"outgoing_receipts_big_size_limit": 999999999999999
},
"witness_config": {
"main_storage_proof_size_soft_limit": 999999999999999,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,7 @@ expression: config_view
"max_length_returned_data": 4194304,
"max_contract_size": 4194304,
"max_transaction_size": 4194304,
"max_receipt_size": 999999999999999,
"max_length_storage_key": 2048,
"max_length_storage_value": 4194304,
"max_promises_per_function_call_action": 1024,
Expand Down Expand Up @@ -233,7 +234,9 @@ expression: config_view
"allowed_shard_outgoing_gas": 9223372036854775807,
"max_tx_gas": 9223372036854775807,
"min_tx_gas": 9223372036854775807,
"reject_tx_congestion_threshold": 1.0
"reject_tx_congestion_threshold": 1.0,
"outgoing_receipts_usual_size_limit": 999999999999999,
"outgoing_receipts_big_size_limit": 999999999999999
},
"witness_config": {
"main_storage_proof_size_soft_limit": 999999999999999,
Expand Down
Loading
Loading