Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit number of accepted transactions based on congestion levels #8877

Closed
1 of 2 tasks
aborg-dev opened this issue Apr 4, 2023 · 3 comments
Closed
1 of 2 tasks

Limit number of accepted transactions based on congestion levels #8877

aborg-dev opened this issue Apr 4, 2023 · 3 comments
Assignees
Labels
A-congestion Work aimed at ensuring good system performance under congestion C-tracking-issue Category: a tracking issue

Comments

@aborg-dev
Copy link
Contributor

aborg-dev commented Apr 4, 2023

This is a tracking issue for work on limiting how many transactions chunk producers will include into the chunk based on the general congestion level in the system.

This is not expected to be a full solution for multi-shard scenario (because of cross-shard incoming receipts), but it would be on a critical path towards providing a cap on the memory usage of delayed receipts and on the wait time for accepted transactions.

Ideas to try

  1. A-congestion A-transaction-runtime S-blocked T-core
@aborg-dev aborg-dev added the A-congestion Work aimed at ensuring good system performance under congestion label Apr 4, 2023
@aborg-dev
Copy link
Contributor Author

Comment from Bowen:
If we want to really enforce this, it will be a protocol change.

We will need to decide whether this capacity is measured in gas or number of receipts. We can start with something simple.

@aborg-dev aborg-dev added the C-tracking-issue Category: a tracking issue label Apr 12, 2023
@aborg-dev
Copy link
Contributor Author

Right now, when a transaction is accepted by a validator it is only forwarded to a few other validators.
The fact that the transaction is present only in a few transaction pools might affect end-to-end transaction processing latency (from the moment the transaction is submitted till the moment it is processed) because the transaction only has a chance of being included in the chunk during the turn of one of those validators and they will not always be able to include all pending transactions to the chunk due to limits we want to introduce, which will effectively delay such transactions.

It would be interesting to see whether this indeed would be the case in the congestion test #8920

@aborg-dev aborg-dev self-assigned this Jun 16, 2023
near-bulldozer bot pushed a commit that referenced this issue Jul 27, 2023
…9222)

This PR introduces a soft limit on the number of delayed receipts in each shard. The limit is around 10000 delayed receipts to make sure this is enough to saturate the chunk capacity even if receipts are very small. The largest number of delayed receipts that we've seen in the last 3 months is around 400 and it stays at 0 most of the time.

In the future, we would be also using other signals like the size of delayed receipts, but this information will have to be first computed and stored in the chunk headers/trie store and will be done in #9228.

Addresses: #8877
@aborg-dev
Copy link
Contributor Author

This is implemented with #9222, now we have a per-shard limit of ~20000 delayed receipts after which chunk producers will stop including more transactions into the chunks that they produce.

nikurt pushed a commit that referenced this issue Jul 28, 2023
…9222)

This PR introduces a soft limit on the number of delayed receipts in each shard. The limit is around 10000 delayed receipts to make sure this is enough to saturate the chunk capacity even if receipts are very small. The largest number of delayed receipts that we've seen in the last 3 months is around 400 and it stays at 0 most of the time.

In the future, we would be also using other signals like the size of delayed receipts, but this information will have to be first computed and stored in the chunk headers/trie store and will be done in #9228.

Addresses: #8877
nikurt pushed a commit that referenced this issue Aug 24, 2023
…9222)

This PR introduces a soft limit on the number of delayed receipts in each shard. The limit is around 10000 delayed receipts to make sure this is enough to saturate the chunk capacity even if receipts are very small. The largest number of delayed receipts that we've seen in the last 3 months is around 400 and it stays at 0 most of the time.

In the future, we would be also using other signals like the size of delayed receipts, but this information will have to be first computed and stored in the chunk headers/trie store and will be done in #9228.

Addresses: #8877
nikurt pushed a commit to nikurt/nearcore that referenced this issue Aug 24, 2023
…ear#9222)

This PR introduces a soft limit on the number of delayed receipts in each shard. The limit is around 10000 delayed receipts to make sure this is enough to saturate the chunk capacity even if receipts are very small. The largest number of delayed receipts that we've seen in the last 3 months is around 400 and it stays at 0 most of the time.

In the future, we would be also using other signals like the size of delayed receipts, but this information will have to be first computed and stored in the chunk headers/trie store and will be done in near#9228.

Addresses: near#8877
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-congestion Work aimed at ensuring good system performance under congestion C-tracking-issue Category: a tracking issue
Projects
None yet
Development

No branches or pull requests

1 participant