Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wip] Correctly purge zero lamport accounts #7013

Closed
wants to merge 41 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
c6bdaac
save
ryoqun Nov 20, 2019
206a098
finally working test
ryoqun Nov 20, 2019
53edd4a
finally working test
ryoqun Nov 20, 2019
091c2ce
finally working test
ryoqun Nov 20, 2019
0c21d4f
finally working test
ryoqun Nov 20, 2019
bdc6cf6
finally working test
ryoqun Nov 20, 2019
0db6827
finally working test
ryoqun Nov 20, 2019
f01c60e
calculdate StorageEntry::count_and_status::count in generate_index
ryoqun Nov 20, 2019
e19b093
check zero lamport accounts
ryoqun Nov 20, 2019
de90b62
check zero lamport accounts
ryoqun Nov 20, 2019
e618b60
Save
ryoqun Nov 21, 2019
6953a55
Double purge_zero_lamport
ryoqun Nov 21, 2019
372aa5f
Move has_accounts_with_zero_lamports
ryoqun Nov 21, 2019
78bb5c9
Promote count value 0 error
ryoqun Nov 21, 2019
719d07d
Remove unused method
ryoqun Nov 21, 2019
22443b8
For debugging
ryoqun Nov 21, 2019
d273b15
rustfmt & clippy
ryoqun Nov 21, 2019
4c589c9
Fix test
ryoqun Nov 21, 2019
bbcfd87
Don't use XXX
ryoqun Nov 21, 2019
e440ec2
Don't use XXX
ryoqun Nov 21, 2019
5a5b879
just save
ryoqun Nov 21, 2019
fe1eb8e
just save
ryoqun Nov 21, 2019
244e194
just save
ryoqun Nov 21, 2019
794f45f
just save
ryoqun Nov 21, 2019
c15e019
just save
ryoqun Nov 21, 2019
1e21ede
just save
ryoqun Nov 21, 2019
354f6f0
just save
ryoqun Nov 22, 2019
28b537f
just save
ryoqun Nov 22, 2019
0a48cc7
just save
ryoqun Nov 22, 2019
be4e3fd
just save
ryoqun Nov 22, 2019
f5c77fa
rustfmt & clippy
ryoqun Nov 22, 2019
27509b7
clean up
ryoqun Nov 22, 2019
0fa3a31
clean up
ryoqun Nov 22, 2019
72fca15
rustfmt & clippy
ryoqun Nov 22, 2019
646679e
Fix test...
ryoqun Nov 22, 2019
525645a
Remove comment...
ryoqun Nov 22, 2019
198be91
backported test_accounts_db_serialize_zero_and_free
ryoqun Nov 26, 2019
b891f59
Make actually backported test pass
ryoqun Nov 26, 2019
8b54fba
rustfmt & clippy
ryoqun Nov 26, 2019
4c62be8
Add debugs
ryoqun Nov 26, 2019
3fc836c
Add failing case comment
ryoqun Nov 26, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions ledger/src/snapshot_utils.rs
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,7 @@ where
}

pub fn add_snapshot<P: AsRef<Path>>(snapshot_path: P, bank: &Bank) -> Result<()> {
trace!("ryoqun saving snapshot");
bank.purge_zero_lamport_accounts();
Copy link
Member Author

@ryoqun ryoqun Nov 22, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: We can reduce the frequency of purge_zero_lamport_accounts invocations for the performance gains because the invocation is completely optional for creating snapshots other than freeing up some memory consumed by zero-lamport accounts.

let slot = bank.slot();
// snapshot_path/slot
Expand Down Expand Up @@ -211,6 +212,7 @@ pub fn bank_from_archive<P: AsRef<Path>>(
snapshot_path: &PathBuf,
snapshot_tar: P,
) -> Result<Bank> {
trace!("ryoqun loading snapshot");
// Untar the snapshot into a temp directory under `snapshot_config.snapshot_path()`
let unpack_dir = tempfile::tempdir_in(snapshot_path)?;
untar_snapshot_in(&snapshot_tar, &unpack_dir)?;
Expand All @@ -224,6 +226,7 @@ pub fn bank_from_archive<P: AsRef<Path>>(
unpacked_accounts_dir,
)?;

bank.purge_zero_lamport_accounts();
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, now we both do purge zero lamport accounts when saving and loading snapshots.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In contract to this, this call cannot be omitted for security reasons.

if !bank.verify_snapshot_bank() {
panic!("Snapshot bank failed to verify");
}
Expand Down
4 changes: 4 additions & 0 deletions runtime/src/accounts.rs
Original file line number Diff line number Diff line change
Expand Up @@ -331,6 +331,10 @@ impl Accounts {
self.accounts_db.verify_hash_internal_state(slot, ancestors)
}

pub fn verify_account_balances(&self, ancestors: &HashMap<Slot, usize>) -> bool {
self.accounts_db.verify_account_balances(ancestors)
}

pub fn load_by_program(
&self,
ancestors: &HashMap<Slot, usize>,
Expand Down
Loading