fix(runtime) - Mitigate the receipt size limit bug #12633
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… bandwidth requests
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #12633 +/- ##
=======================================
Coverage 70.46% 70.46%
=======================================
Files 845 845
Lines 172287 172308 +21
Branches 172287 172308 +21
=======================================
+ Hits 121396 121418 +22
+ Misses 45779 45776 -3
- Partials 5112 5114 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
wacban
approved these changes
Dec 23, 2024
| /// NewReceipt validation. Tolerates some receipts that wouldn't pass new validation. It has to | ||
| /// be less strict because: | ||
| /// 1) Older receipts might have been created before new validation rules. | ||
| /// 2) There is a bug which allows to create receipts that are above the size limit. Runtime has |
There was a problem hiding this comment.
I thought it's not possible to create them above the size limit but that the receipt may increase in size later?
| /// 2) There is a bug which allows to create receipts that are above the size limit. Runtime has | ||
| /// to handle them gracefully until the receipt size limit bug is fixed. | ||
| /// See https://github.com/near/nearcore/issues/12606 for details. | ||
| OldReceipt, |
There was a problem hiding this comment.
- I can see that the delayed and incoming receipts are validated but the buffered receipts are not, as far as I can see. Is it an omission?
- Is the validation for old receipts actually necessary or is it just an extra protection? Those receipts are a result of on chain computation so I would assume we trust them here already.
Comment on lines
+402
to
+403
| if size > max_receipt_size { | ||
| if size > max_receipt_size { |
There was a problem hiding this comment.
this condition got duplicated
| if size > max_receipt_size { | ||
| tracing::warn!( | ||
| target: "runtime", | ||
| "try_forward observed a receipt with size exceeding the size limit! receipt_id: {} size: {} size_limit: {}", |
There was a problem hiding this comment.
nit: use structured logging - set the values as fields rather than in the formatted message
tracing::warn!(
receipt_id=?..
size,
max_size,
"try_forward observed a receipt with size exceeding the size limit"
);
|
|
||
| // A function call will generate a new receipt. Size of this receipt will be equal to | ||
| // `max_receipt_size`, it'll pass validation, but then `output_data_receivers` will be modified and | ||
| // the receipt's size will go above max_receipt_size. The receipt should be rejected, but currently |
There was a problem hiding this comment.
Can you assert that this condition is actually triggered?
Comment on lines
+1787
to
+1788
| /// Returns a value of size "value_size". | ||
| /// Accepts json args, e.g {"value_size": 1000} |
There was a problem hiding this comment.
just out of curiosity, why not pass it as simple integer instead of the json?
Comment on lines
+7
to
+8
| #[allow(unused)] | ||
| extern "C" { |
There was a problem hiding this comment.
- Is all of it needed?
- Is there any way we can reuse the existing one or refactor it to avoid copy pasting?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There is a bug (#12606) which allows to create receipts which are slightly larger than
max_receipt_size.Let's mitigate the issue by making the runtime able to handle receipts that are above
max_receipt_size.First the size limit check is relaxed to check only new receipts. Then the receipt forwarding logic is modified to pretend that all receipts have
congestion_sizebelowmax_receipt_size.This mostly matches the code that was added in
2.3.1and2.4.0, but this PR contains one more change that is needed for bandwidth scheduler. Before bandwidth scheduler we were able to forward receipts that are slightly above themax_receipt_sizebecause the big outgoing limit was 4.5 MiB andtry_forwardpretended that all receipts are at most 4 MiB. With bandwidth scheduler this changes, bandwidth scheduler expects all receipts to be at mostmax_receipt_size, otherwise there's no guarantee that the receipt will ever be forwarded. To deal with this there's the change which pretends that all receipts are at mostmax_receipt_sizewhen generating bandwidth requests.This is just a mitigation, a proper fix will be done in the future.
One more advantage of this patch is that it should allow us to reduce
max_receipt_sizewithout crazy protocol upgrade logic. We could reduce it to e.g 2MB and the scheduler would still be able to handle 4MB receipts created in the previous protocol version.I added some of tests which try to create a receipt above max size and check that everything still works fine.
As usual the
test_transaction_limit_for_local_congestiontest broke when I added more methods to thetest-contract-rscontract. For some reason this test is very sensitive to contract size and the congestion level rapidly drops when size of the contract increases. To deal with this I added a separate small contract that is used exclusively by these congestion control tests, that should fix the issue once and for all.The change is divided into commits for easier review.