feat: enable receipt prefetching by default #7661
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@mm-near nominating you as a reviewer since you already reviewed the feature, this one is just the decision to activate it by default from now on |
mm-near
approved these changes
Sep 22, 2022
Prefetch receipt meta data (account and access keys) ahead of time. This recent performance optimization has been disabled by default. In lab settings, performance improvement is confirmed. Using the estimator to measure the time it takes to process empty receipts, on a DB with 2 million accounts, on a local SSD, with enabled shard caches. The result is as follows. - sender = receiver: 737us -> 386 us - sender != receiver: 1014us -> 644us - overhead per block: 6.9us -> 7.4us Note that this is with 100 empty receipts in the same block, with all different accounts. In real traffic it usually does not happen that so many different accounts are accessed in the same block. But it is allowed and we must be able process this case in reasonable time. So even if it might not help in the average case, it makes sense to activate this feature to speed up the worst-case. Currently we use 8 IO threads per shard. Repeated experiments with more threads showed no difference. Decreasing it to 4 threads is about equal to 8 threads. Going lower is significantly worse. Thus, overall, 8 threads seems reasonable here. Canary nodes in testnet and mainnet with the feature enabled show that the feature also works as expected on real traffic. The memory impact is minimal, usually less than 40MB of reserved capacity, which is less than 8MB actual memory because 8 threads reserve 4MB each ahead of actually fetching the data.
jakmeier
force-pushed
the
enable-receipt-prefetch
branch
from
9e711b2
to
d98d7b9
Compare
jakmeier
added a commit
to jakmeier/nearcore
that referenced
this pull request
Sep 26, 2022
Explicitly stop and wait for prefetching background threads to terminate when a testbed is dropped. This avoids that estimations are influenced by background threads left over from previous estimations, which we have observed since merging near#7661.
nikurt
pushed a commit
that referenced
this pull request
Sep 26, 2022
Prefetch receipt meta data (account and access keys) ahead of time. This recent performance optimization has been disabled by default. In lab settings, performance improvement is confirmed. Using the estimator to measure the time it takes to process empty receipts, on a DB with 2 million accounts, on a local SSD, with enabled shard caches. The result is as follows. - sender = receiver: 737us -> 386 us - sender != receiver: 1014us -> 644us - overhead per block: 6.9us -> 7.4us Note that this is with 100 empty receipts in the same block, with all different accounts. In real traffic it usually does not happen that so many different accounts are accessed in the same block. But it is allowed and we must be able process this case in reasonable time. So even if it might not help in the average case, it makes sense to activate this feature to speed up the worst-case. Currently we use 8 IO threads per shard. Repeated experiments with more threads showed no difference. Decreasing it to 4 threads is about equal to 8 threads. Going lower is significantly worse. Thus, overall, 8 threads seems reasonable here. Canary nodes in testnet and mainnet with the feature enabled show that the feature also works as expected on real traffic. The memory impact is minimal, usually less than 40MB of reserved capacity, which is less than 8MB actual memory because 8 threads reserve 4MB each ahead of actually fetching the data.
jakmeier
added a commit
to jakmeier/nearcore
that referenced
this pull request
Sep 28, 2022
Explicitly stop and wait for prefetching background threads to terminate when the `ShardTriesInner` is dropped. This avoids that estimations are influenced by background threads left over from previous estimations, which we have observed since merging near#7661.
near-bulldozer bot
pushed a commit
that referenced
this pull request
Sep 29, 2022
Explicitly stop and wait for prefetching background threads to terminate when the `ShardTriesInner` is dropped. This avoids that estimations are influenced by background threads left over from previous estimations, which we have observed since merging #7661.
nikurt
pushed a commit
that referenced
this pull request
Nov 9, 2022
Prefetch receipt meta data (account and access keys) ahead of time. This recent performance optimization has been disabled by default. In lab settings, performance improvement is confirmed. Using the estimator to measure the time it takes to process empty receipts, on a DB with 2 million accounts, on a local SSD, with enabled shard caches. The result is as follows. - sender = receiver: 737us -> 386 us - sender != receiver: 1014us -> 644us - overhead per block: 6.9us -> 7.4us Note that this is with 100 empty receipts in the same block, with all different accounts. In real traffic it usually does not happen that so many different accounts are accessed in the same block. But it is allowed and we must be able process this case in reasonable time. So even if it might not help in the average case, it makes sense to activate this feature to speed up the worst-case. Currently we use 8 IO threads per shard. Repeated experiments with more threads showed no difference. Decreasing it to 4 threads is about equal to 8 threads. Going lower is significantly worse. Thus, overall, 8 threads seems reasonable here. Canary nodes in testnet and mainnet with the feature enabled show that the feature also works as expected on real traffic. The memory impact is minimal, usually less than 40MB of reserved capacity, which is less than 8MB actual memory because 8 threads reserve 4MB each ahead of actually fetching the data.
nikurt
pushed a commit
that referenced
this pull request
Nov 9, 2022
Explicitly stop and wait for prefetching background threads to terminate when the `ShardTriesInner` is dropped. This avoids that estimations are influenced by background threads left over from previous estimations, which we have observed since merging #7661.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Prefetch receipt meta data (account and access keys) ahead of time.
This recent performance optimization has been disabled by default.
In lab settings, performance improvement is confirmed. Using the
estimator to measure the time it takes to process empty receipts,
on a DB with 2 million accounts, on a local SSD, with enabled shard
caches. The result is as follows.
Note that this is with 100 empty receipts in the same block, with all
different accounts. In real traffic it usually does not happen that so
many different accounts are accessed in the same block. But it is
allowed and we must be able process this case in reasonable time.
So even if it might not help in the average case, it makes sense to
activate this feature to speed up the worst-case.
Currently we use 8 IO threads per shard. Repeated experiments with
more threads showed no difference.
Decreasing it to 4 threads is about equal to 8 threads. Going lower is
significantly worse. Thus, overall, 8 threads seems reasonable here.
Canary nodes in testnet and mainnet with the feature enabled show that
the feature also works as expected on real traffic. The memory impact is
minimal, usually less than 40MB of reserved capacity, which is less than
8MB actual memory because 8 threads reserve 4MB each ahead of actually
fetching the data.