-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Retry hash file allocation #33565
Retry hash file allocation #33565
Conversation
I'm not familiar with this part of the kernel. How likely is it that (1) fragmentation was the issue, and (2) that a defrag will run before we've hit our retry limit? |
Is there any way to test this? I can't think of anything deterministic off the top of my head. But that would be nice... |
Codecov Report
@@ Coverage Diff @@
## master #33565 +/- ##
=========================================
- Coverage 81.7% 81.7% -0.1%
=========================================
Files 807 807
Lines 218287 218305 +18
=========================================
- Hits 178524 178501 -23
- Misses 39763 39804 +41 |
37842ba
to
d287913
Compare
This seems reasonable to me. I'd like to see a data point published if we actually have to retry. |
2a5a57f
to
5a20ee7
Compare
accounts-db/src/accounts_hash.rs
Outdated
data.write_all(&[0]).unwrap(); | ||
data.rewind().unwrap(); | ||
data.flush().unwrap(); | ||
// Retry 5 times for allocation the AccountHashFile. The memory maybe fragmented and |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
typo
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you mean AccountHashesFile?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Original:
Retry 5 times for allocation the..
suggestion:
Retry 5 times to allocate the...
Maybe more grammar than typo.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok. done.
accounts-db/src/accounts_hash.rs
Outdated
num_retries | ||
); | ||
} | ||
datapoint_info!("account_hash_file_allocation_retry", ("retry", 1, i64),); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
trailing , to remove since there's another typo above.
d31617f
to
42245e1
Compare
); | ||
} | ||
datapoint_info!("retry_account_hashes_file_allocation", ("retry", 1, i64)); | ||
thread::sleep(time::Duration::from_millis(num_retries * 100)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
100 milliseconds is a long time. Maybe too long? I'm not sure. How was this constant chosen?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah. I think this should not happen very often, so choose 100ms to be conservative.
Co-authored-by: Brooks <brooks@prumo.org>
It seems that a similar crash happened again on mc3 last Friday. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
* retry hash file allocation * add sleep * submit a datapoint for retry * typo * more typos * Update accounts-db/src/accounts_hash.rs Co-authored-by: Brooks <brooks@prumo.org> * fmt --------- Co-authored-by: HaoranYi <haoran.yi@solana.com> Co-authored-by: Brooks <brooks@prumo.org> (cherry picked from commit 167dac2) # Conflicts: # accounts-db/src/accounts_hash.rs
* Retry hash file allocation (#33565) * retry hash file allocation * add sleep * submit a datapoint for retry * typo * more typos * Update accounts-db/src/accounts_hash.rs Co-authored-by: Brooks <brooks@prumo.org> * fmt --------- Co-authored-by: HaoranYi <haoran.yi@solana.com> Co-authored-by: Brooks <brooks@prumo.org> (cherry picked from commit 167dac2) # Conflicts: # accounts-db/src/accounts_hash.rs * fix conflicts --------- Co-authored-by: HaoranYi <haoran.yi@gmail.com> Co-authored-by: HaoranYi <haoran.yi@solana.com>
…lana-labs#33918) * Retry hash file allocation (solana-labs#33565) * retry hash file allocation * add sleep * submit a datapoint for retry * typo * more typos * Update accounts-db/src/accounts_hash.rs Co-authored-by: Brooks <brooks@prumo.org> * fmt --------- Co-authored-by: HaoranYi <haoran.yi@solana.com> Co-authored-by: Brooks <brooks@prumo.org> (cherry picked from commit 167dac2) # Conflicts: # accounts-db/src/accounts_hash.rs * fix conflicts --------- Co-authored-by: HaoranYi <haoran.yi@gmail.com> Co-authored-by: HaoranYi <haoran.yi@solana.com>
…lana-labs#33918) * Retry hash file allocation (solana-labs#33565) * retry hash file allocation * add sleep * submit a datapoint for retry * typo * more typos * Update accounts-db/src/accounts_hash.rs Co-authored-by: Brooks <brooks@prumo.org> * fmt --------- Co-authored-by: HaoranYi <haoran.yi@solana.com> Co-authored-by: Brooks <brooks@prumo.org> (cherry picked from commit 167dac2) # Conflicts: # accounts-db/src/accounts_hash.rs * fix conflicts --------- Co-authored-by: HaoranYi <haoran.yi@gmail.com> Co-authored-by: HaoranYi <haoran.yi@solana.com>
Problem
Recently, on Oct 6, 20023, we saw a validator crash on one of the canaries boxes.
The crash happens when allocating account hash file during hash dedup.
The
used
memory increase by 2G and the free memory drops to 0.65 percent.It appears that the box still has enough physical memory. However, the allocation fails. Probably due to memory defragmentation and multiple threads allocate mmaps at the same time.
Summary of Changes
Account hash calculation is highly parallel. It seems that when all threads start allocation files to store account hashes at the same time, the system may hit OOM for some of those allocations. To mitigate that, we add retry for account hash file allocation. If a thread fails its allocation, it will sleep for some time and then retry. This will help to stagger the memory allocation from all parallel threads.
Hopefully, the kernel has time to defrag the memory in-between the retries, and the later retry for allocation will succeed.
Fixes #