Remove archiver and storage program #9992

jackcmay · 2020-05-11T23:47:24Z

Problem

Archiver and storage program not used and are a potential security risk

Summary of Changes

Remove them

Fixes #

jackcmay · 2020-05-11T23:52:47Z

@sakridge Couple loose ends:

archiver-utils, it's used in chacha-cuda tests, are those test still required?
Are segments still required as defined in sdk/src/clock.rs?
Sysvar Rewards includes storage rewards, we probably can't remove them but maybe just rename them as unused, thoughts?
Do we still need this:

solana/core/src/repair_service.rs

Line 152 in 3b9dc50

// Strategy used by archivers
Do we still want to ignore the send error here:

solana/core/src/window_service.rs

Line 208 in 3b9dc50

// Ignore the send error, as the retransmit is optional (e.g. archivers don't retransmit)

codecov · 2020-05-12T01:08:31Z

Codecov Report

Merging #9992 into master will increase coverage by 1.1%.
The diff coverage is 92.4%.

@@           Coverage Diff            @@
##           master   #9992     +/-   ##
========================================
+ Coverage    80.5%   81.7%   +1.1%     
========================================
  Files         295     281     -14     
  Lines       68157   65625   -2532     
========================================
- Hits        54928   53645   -1283     
+ Misses      13229   11980   -1249

sakridge · 2020-05-12T05:10:15Z

@sakridge Couple loose ends:

* archiver-utils, it's used in chacha-cuda tests, are those test still required?

Believe those can all be removed. chacha/chacha-cuda/chacha-sys

Are segments still required as defined in sdk/src/clock.rs?

Pretty sure it can be removed.

Rewards

Rename as deprecated sounds good.

Repair strategy.

Probably can be removed @carllin ?

Send error.

Hmm, not sure. In theory we could return the error there. Can try to return it and see if it works. Or leaving it for now is probably fine too.

sakridge · 2020-05-12T05:20:48Z

solana/perf/src/perf_libs.rs

Line 60 in 84b6120

pub chacha_cbc_encrypt_many_sample: Symbol<

These definitions can also be removed: chacha_cbc_encrypt_many_sample, chacha_init_sha_state, chacha_end_sha_state.

jackcmay · 2020-05-12T05:21:47Z

@sakridge to confirm, you mean wholesale removal of chacha, chacha-cuda, and chacha-sys?

sakridge · 2020-05-12T05:22:03Z

@sakridge to confirm, you mean wholesale removal of chacha, chacha-cuda, and chacha-sys?

yep

runtime/src/bank.rs

cli/src/cli.rs

jackcmay · 2020-05-13T16:53:06Z

@carllin Do we need this once archiver's and storage stage are removed?

solana/core/src/repair_service.rs

Line 152 in 3b9dc50

// Strategy used by archivers

carllin · 2020-05-13T18:26:39Z

@carllin Do we need this once archiver's and storage stage are removed?

solana/core/src/repair_service.rs

Line 152 in 3b9dc50

// Strategy used by archivers

Nope they can be removed!

jackcmay · 2020-05-13T18:51:31Z

@carllin Then should I also pull out this test since it used the RepairRange strategy?

solana/core/src/window_service.rs

Line 654 in 13bc3f8

fn test_recv_window() {

carllin · 2020-05-13T19:10:15Z

@jackcmay I've been looking at that test, it was added here: https://github.com/solana-labs/solana/pull/5804/files

..and I'm not sure what it's really even testing 😛

So I say we can just remove it 😃

jackcmay · 2020-05-13T20:21:19Z

core/src/repair_service.rs

+    pub bank_forks: Arc<RwLock<BankForks>>,
+    pub completed_slots_receiver: CompletedSlotsReceiver,
+    pub epoch_schedule: EpochSchedule,
+    pub duplicate_slots_reset_sender: DuplicateSlotsResetSender,
 }


@carllin Removed and replaced with a single strategy

Awesome, looks great, thanks!

jackcmay · 2020-05-13T22:15:59Z

@sakridge @mvines @danpaul000 Any objections before I commit this?

danpaul000

Just looked over net/ scripts, no objection there.

sakridge

looks ok to me

jackcmay · 2020-05-14T00:20:15Z

runtime/src/bank.rs

@@ -2063,11 +2058,6 @@ impl Bank {

        if Stakes::is_stake(account) {
            self.stakes.write().unwrap().store(pubkey, account);
-        } else if storage_utils::is_storage(account) {


@sakridge this will cause a snapshot to fail?

Don't believe so, because these fields are not covered by any hash in the snapshot. As long as it deserializes the same it should be ok.

runtime/src/bank.rs

ryoqun · 2020-05-14T02:34:31Z

runtime/src/bank.rs

@@ -595,7 +606,7 @@ impl Bank {
    pub fn clock(&self) -> sysvar::clock::Clock {
        sysvar::clock::Clock {
            slot: self.slot,
-            segment: get_segment_from_slot(self.slot, self.slots_per_segment),
+            unused: 0,


@jackcmay Oops, I think this might affect bankhash... Well, segment is ensured to be 0 for all cases and times on tds and mainnet-beta?

How is the clock returned by this function used in the bankhash?

Sorry for late reply..

Firstly, all of sysvars are now stored in AccountsDB meaning they're part of bankhash, which wasn't case before #7892.

Then, the returned value from this function is used at here: https://github.com/solana-labs/solana/pull/9992/files#diff-a7549f152920d85fb44e6a784b8e5e1fR612

Then, the sysvar account is serialized according to this, which isn't #[serde(skip)],
https://github.com/solana-labs/solana/pull/9992/files#diff-817709fb5feaff9180439b7ff6369730R84.

So, I think this needs some special handling unless segment had been always 0 in tds/mainnet-beta.

Fortunately, I think the migration path is easy to do: Just create a small prs for v1.0 and v1.1 which starts to return segment: 0 at specific operatingmode/slot/epoch whatever. :)

I was able to load the mainnet-beta snapshot and run a validator. Let me know what other tests you think might catch what you suspect.

@jackcmay Well, I was able to produce errors by manual. Anyway, I think this is too hard to verify by hand unless you're snapshot expert.... So, we definitely need to add test...

In short, I can produce an critical error indicating a permenant fork from the cluster:

[2020-05-15T04:56:50.063881235Z WARN solana_vote_program::vote_state] GdnSyH3YtwcxFvQrVVJMm1JhTS4QVX7MFsX56uJLUfiZ dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz

These messages like dropped vote ... failed to match HASH indicates the validator forked from the others in the cluster due to mismatched bankhashes, which is bad.

TEST SETUP

Sadly, I've landed other bankhash-affecting prs, so we need to revert the prs or patch to effectively disable those changes like this:

commit 9a1c021bac2d027fe0a9b237e069ea6d7745cd86 (HEAD -> master) Author: Ryo Onodera <ryoqun@gmail.com> Date: Fri May 15 14:34:58 2020 +0900 Be compatible with mainnet-beta by reverting diff --git a/runtime/src/accounts_db.rs b/runtime/src/accounts_db.rs index 61cf9a88f..da3be4089 100644 --- a/runtime/src/accounts_db.rs +++ b/runtime/src/accounts_db.rs @@ -1425,7 +1425,7 @@ impl AccountsDB { hasher.hash(&[0u8; 1]); } - hasher.hash(&owner.as_ref()); + //hasher.hash(&owner.as_ref()); hasher.hash(&pubkey.as_ref()); hasher.result() diff --git a/runtime/src/bank.rs b/runtime/src/bank.rs index 5b78afb4b..6b6acf530 100644 --- a/runtime/src/bank.rs +++ b/runtime/src/bank.rs @@ -840,7 +840,7 @@ impl Bank { if *hash == Hash::default() { // finish up any deferred changes to account state - self.collect_rent_eagerly(); // update the docs + //self.collect_rent_eagerly(); // update the docs self.collect_fees(); self.distribute_rent(); self.update_slot_history();

And git graph like this:

$ git log -n 10 --graph --oneline a914bc5b72684229f5df96315495c9af7df39e45 9a1c021ba * 9a1c021ba (HEAD -> master) Be compatible with mainnet-beta by reverting * eb1acaf92 (origin/master, origin/HEAD) Remove archiver and storage program (#9992) | * a914bc5b7 Be compatible with mainnet-beta by reverting |/ * 9ef9969d2 Remove notifier module duplication (#10051) * 40b7c1126 Base58 (#10052) * d195dce5d (CriesofCarrots/master) Clean up Ledger instructions (#10047) * 816bf6ebd Bump fnv from 1.0.6 to 1.0.7 (#10046) * ed53a70b5 Cli: transfer ALL; check spend+fee in client (#10012) * 4e4a21f9b (danpaul000/master) `solana-gossip spy` can now specify a shred version (#10040) * c5460e7fe Remove inline from all BPF C functions (#10038)

I tested at a914bc5b7 (GOOD; where error doesn't occur) and 9a1c021ba (BAD; where error does occur).

TEST STEP

Assuming you have recent snapshot and ledger (which I describe after this section), you can reproduce this error just by running ledger-tool verify like this:

Good (at a914bc5b7)

NO dropped vote ... failed to match HASH errors.

$ solana-ledger-tool --version solana-ledger-tool 1.2.0 a914bc5b7 $ solana-ledger-tool --ledger /tmp/mainnet-beta-archive-removal/ verify genesis hash: 5eykt4UsFv8P8NJdTREpY1vzqKqZKvdpKuc147dw2N9d [2020-05-15T05:38:24.325311924Z INFO solana_ledger::blockstore] Maximum open file descriptors: 65000 [2020-05-15T05:38:24.325365021Z INFO solana_ledger::blockstore] Opening database at "/tmp/mainnet-beta-archive-removal/rocksdb" [2020-05-15T05:38:24.341897123Z INFO solana_ledger::blockstore] "/tmp/mainnet-beta-archive-removal/rocksdb" open took 16ms [2020-05-15T05:38:24.341988104Z INFO solana_ledger::bank_forks_utils] Initializing snapshot path: "/tmp/mainnet-beta-archive-removal/snapshot" [2020-05-15T05:38:24.347607486Z INFO solana_ledger::bank_forks_utils] Loading snapshot package: "/tmp/mainnet-beta-archive-removal/snapshot-11079842-G3twNav4jRDVv1nLZyVSd4nALMRXEDRf4g9Jqg8PgZNJ.tar.bz2" [2020-05-15T05:38:53.306619933Z INFO solana_ledger::snapshot_utils] snapshot untar took 29.0s [2020-05-15T05:38:53.306719788Z INFO solana_ledger::snapshot_utils] snapshot version: 1.1.0 [2020-05-15T05:38:53.306809438Z INFO solana_ledger::snapshot_utils] Loading bank from "/tmp/mainnet-beta-archive-removal/snapshot/.tmp7QizfK/snapshots/11079842/11079842" [2020-05-15T05:38:53.370837167Z INFO solana_ledger::snapshot_utils] Rebuilding accounts... [2020-05-15T05:41:49.945345159Z INFO solana_ledger::snapshot_utils] Rebuilding status cache... [2020-05-15T05:41:50.350681494Z INFO solana_ledger::snapshot_utils] Loaded bank for slot: 11079842 [2020-05-15T05:41:50.352265110Z INFO solana_runtime::accounts_db] total_stores: 2608, newest_slot: 11079842, oldest_slot: 0, max_slot: 0 (num=5), min_slot: 4718744 (num=1) [2020-05-15T05:41:50.352384199Z INFO solana_metrics::metrics] metrics disabled: SOLANA_METRICS_CONFIG: environment variable not found [2020-05-15T05:41:50.352686546Z INFO solana_metrics::metrics] datapoint: accounts_db-stores total_count=2608i [2020-05-15T05:41:50.352876531Z INFO solana_metrics::metrics] datapoint: accounts_db-perf-stats delta_hash_num=0i delta_hash_scan_us=0i delta_hash_merge_us=0i delta_hash_accumulate_us=0i [2020-05-15T05:41:54.166416330Z INFO solana_runtime::bank] bank frozen: 11079842 hash: 9Aj3XorHCeVk1JKiV1vKjBqVB8FRiJze3gPx2FqohT2g accounts_delta: EjR393U12AeojVPBVCAjAjgjhhz5Mix7ce4JgnsAB5Za signature_count: 61 last_blockhash: zru76SK5dQqeZQd9VEZt8bTzPWSi2MXSb5VZvc8QTNj capitalization: 499996862353074280 [2020-05-15T05:41:54.166761103Z INFO solana_runtime::bank] accounts hash slot: 11079842 stats: BankHashStats { num_updated_accounts: 128, num_removed_accounts: 0, num_lamports_stored: 25024604877564, total_data_len: 385232, num_executable_accounts: 0 } [2020-05-15T05:41:54.166839787Z INFO solana_ledger::snapshot_utils] bank rebuild from snapshot took 180.9s [2020-05-15T05:41:54.169723674Z INFO solana_ledger::blockstore_processor] processing ledger from slot 11079842... [2020-05-15T05:41:54.194129216Z INFO solana_ledger::blockstore_processor] ledger holds data through slot 11081264 [2020-05-15T05:41:55.783404680Z INFO solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=11079843i block_height=10730179i [2020-05-15T05:42:00.169123981Z INFO solana_runtime::bank] bank frozen: 11079843 hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP accounts_delta: 9gAMaX1o9gzK2sCSWVsg9PHtH8HrsDAhjNW1XDu2omcx signature_count: 96 last_blockhash: CX98iQ5TxS7QH9dfTW9x6r1NSkBkAcKy9kJvJm8wKf1j capitalization: 499996862352594280 [2020-05-15T05:42:00.169471635Z INFO solana_runtime::bank] accounts hash slot: 11079843 stats: BankHashStats { num_updated_accounts: 198, num_removed_accounts: 0, num_lamports_stored: 37404817053384, total_data_len: 515817, num_executable_accounts: 0 } [2020-05-15T05:42:00.171866513Z INFO solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=11079844i block_height=10730180i [2020-05-15T05:42:00.185213631Z INFO solana_ledger::blockstore_processor] processing ledger: slot=11079844, last root slot=11079842 slots=1 slots/s=0.25 txs/s=24 [2020-05-15T05:42:04.087799607Z INFO solana_runtime::bank] bank frozen: 11079844 hash: 4oNAwZxvcHMbSeqRqoG9ra4LxRtvE2uo1e2yeHgjL4i9 accounts_delta: 76Hv7Bq8Dbdh29QTF7PbtgHi7X6WErRR4pkyfr9ayd1H signature_count: 85 last_blockhash: BeVeHQHvVZNPrHkbnw7iAMDysmTq42dQ88g44Rj1WRoQ capitalization: 499996862352169280 [2020-05-15T05:42:04.088140151Z INFO solana_runtime::bank] accounts hash slot: 11079844 stats: BankHashStats { num_updated_accounts: 170, num_removed_accounts: 0, num_lamports_stored: 34201303008164, total_data_len: 452390, num_executable_accounts: 0 } [2020-05-15T05:42:04.090744458Z INFO solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=11079845i block_height=10730181i [2020-05-15T05:42:04.104299548Z INFO solana_ledger::blockstore_processor] processing ledger: slot=11079845, last root slot=11079842 slots=1 slots/s=0.33333334 txs/s=28.333334 [2020-05-15T05:42:07.612519971Z INFO solana_runtime::bank] bank frozen: 11079845 hash: 4dyvZqbQ5fiDa6c1WUrs152mqq38Ve1t7ZjLj5eNtMMW accounts_delta: 4Gk1PA6KUqH28KM8u294BczSUeZ32QyFUkbTwQ7faXVV signature_count: 33 last_blockhash: 4reo5wDHFe3nhtUXT33zXCVMwFs8F8nFZaEiCGxZuroL capitalization: 499996862352004280 [2020-05-15T05:42:07.612868105Z INFO solana_runtime::bank] accounts hash slot: 11079845 stats: BankHashStats { num_updated_accounts: 72, num_removed_accounts: 0, num_lamports_stored: 13457574056164, total_data_len: 280764, num_executable_accounts: 0 } [2020-05-15T05:42:07.615450325Z INFO solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=11079846i block_height=10730182i [2020-05-15T05:42:07.628964945Z INFO solana_ledger::blockstore_processor] processing ledger: slot=11079846, last root slot=11079842 slots=1 slots/s=0.33333334 txs/s=11 [2020-05-15T05:42:11.750445835Z INFO solana_runtime::bank] bank frozen: 11079846 hash: CZ1ediuKEQzowCgB1L8KRDQM1UyMXfx421u9M9nQBn1K accounts_delta: 6W9fxESp7kLDULoLFeDpZnCN1DamVHdnotAw8AaxpXf6 signature_count: 49 last_blockhash: HJhMEypFNR3oWbAzDKB7QyFCEqLDZ9r3pGDQtXZKKsAY capitalization: 499996862351759280 [2020-05-15T05:42:11.750790099Z INFO solana_runtime::bank] accounts hash slot: 11079846 stats: BankHashStats { num_updated_accounts: 104, num_removed_accounts: 0, num_lamports_stored: 20108395046905, total_data_len: 340460, num_executable_accounts: 0 } [2020-05-15T05:42:11.753437002Z INFO solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=11079847i block_height=10730183i [2020-05-15T05:42:11.766785416Z INFO solana_ledger::blockstore_processor] processing ledger: slot=11079847, last root slot=11079842 slots=1 slots/s=0.25 txs/s=12.25 [2020-05-15T05:42:15.377300199Z INFO solana_runtime::bank] bank frozen: 11079847 hash: jMuMWFRy3whioLnxxN3hKhKe8N3FGpQtHn6jfwnErQD accounts_delta: 4iVj7GoYFu8haZUmi1wvAZmoVc5HLFgwrUhiEQkDtVkQ signature_count: 19 last_blockhash: FfFYu4VpyQNAxzmVq1DxMEyaFn41KSdWemfbxsa73pr1 capitalization: 499996862351664280 [2020-05-15T05:42:15.377648574Z INFO solana_runtime::bank] accounts hash slot: 11079847 stats: BankHashStats { num_updated_accounts: 43, num_removed_accounts: 0, num_lamports_stored: 8333738896405, total_data_len: 224799, num_executable_accounts: 0 } [2020-05-15T05:42:15.380398471Z INFO solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=11079848i block_height=10730184i [2020-05-15T05:42:15.393784388Z INFO solana_ledger::blockstore_processor] processing ledger: slot=11079848, last root slot=11079842 slots=1 slots/s=0.33333334 txs/s=6.3333335 [2020-05-15T05:42:20.336072515Z INFO solana_runtime::bank] bank frozen: 11079848 hash: CSqxc69wnuwoD8qjGHDY3rz5R1XGmVLT7uKz2tx3RmU2 accounts_delta: 36hvTdrXW12XdUXztUKgPYweDoPaRhe9MxW6N28gL5Vg signature_count: 227 last_blockhash: 3dFQRc3Q96Af2V1Z9htemgoQadrAh6JzATnmStsWaLMn capitalization: 499996862350529280 [2020-05-15T05:42:20.336431258Z INFO solana_runtime::bank] accounts hash slot: 11079848 stats: BankHashStats { num_updated_accounts: 420, num_removed_accounts: 0, num_lamports_stored: 88541262360302, total_data_len: 855338, num_executable_accounts: 0 } [2020-05-15T05:42:20.338896691Z INFO solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=11079849i block_height=10730185i [2020-05-15T05:42:20.352429687Z INFO solana_ledger::blockstore_processor] processing ledger: slot=11079849, last root slot=11079842 slots=1 slots/s=0.25 txs/s=56.75 [2020-05-15T05:42:23.958691262Z INFO solana_runtime::bank] bank frozen: 11079849 hash: ADaoqupwoQ3DPp9euQpGetAhsL6rw8gH1sbWnGzN3iV5 accounts_delta: FUqthsELZZjwAba9nW9bicmLaijNsB18Nu4mtRpypWrL signature_count: 15 last_blockhash: 414qufZyFnfXsE5KaZcT8DN56ETELMrJ2NUSAS8jhW9K capitalization: 499996862350454280 [2020-05-15T05:42:23.959210579Z INFO solana_runtime::bank] accounts hash slot: 11079849 stats: BankHashStats { num_updated_accounts: 34, num_removed_accounts: 0, num_lamports_stored: 5894470161085, total_data_len: 206144, num_executable_accounts: 0 } [2020-05-15T05:42:23.961306733Z INFO solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=11079850i block_height=10730186i [2020-05-15T05:42:23.975138819Z INFO solana_ledger::blockstore_processor] processing ledger: slot=11079850, last root slot=11079842 slots=1 slots/s=0.33333334 txs/s=5 [2020-05-15T05:42:27.975392756Z INFO solana_runtime::bank] bank frozen: 11079850 hash: ADVeR4mr48i8WFpmQyBU6svFyibqJoznGU8Qqi48znUn accounts_delta: bxYHeVy34GBDMruthkAqg3NCfDMXJAc5U3XAH38EHCc signature_count: 93 last_blockhash: 6UPe8XQreYp11fUkXJuw7vMLjQrNKsj5jVWxYoFK5reT capitalization: 499996862349989280 [2020-05-15T05:42:27.975702487Z INFO solana_runtime::bank] accounts hash slot: 11079850 stats: BankHashStats { num_updated_accounts: 190, num_removed_accounts: 0, num_lamports_stored: 37804700067503, total_data_len: 497162, num_executable_accounts: 0 } [2020-05-15T05:42:27.977740684Z INFO solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=11079851i block_height=10730187i [2020-05-15T05:42:27.990887268Z INFO solana_ledger::blockstore_processor] processing ledger: slot=11079851, last root slot=11079842 slots=1 slots/s=0.25 txs/s=23.25 [2020-05-15T05:42:31.701257082Z INFO solana_runtime::bank] bank frozen: 11079851 hash: 4HUx3dFpzSWzAEYAm3mU69gEwiRacvubEuMr4uZ8Dyzh accounts_delta: 6iJyJ5M2vdo96j5HuQyAUhGxUCkDZoW4mBjnGBzpYjzG signature_count: 67 last_blockhash: DoEUbKME8dfzYdmrZak272hA1ga3nRc1zWdq6S1syH7A capitalization: 499996862349654280 [2020-05-15T05:42:31.701600140Z INFO solana_runtime::bank] accounts hash slot: 11079851 stats: BankHashStats { num_updated_accounts: 140, num_removed_accounts: 0, num_lamports_stored: 26038873757965, total_data_len: 407618, num_executable_accounts: 0 } [2020-05-15T05:42:31.704085662Z INFO solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=11079852i block_height=10730188i [2020-05-15T05:42:31.717219710Z INFO solana_ledger::blockstore_processor] processing ledger: slot=11079852, last root slot=11079842 slots=1 slots/s=0.33333334 txs/s=22.333334

Bad (at 9a1c021ba)

MANY dropped vote ... failed to match HASH errors.

$ solana-ledger-tool --version solana-ledger-tool 1.2.0 9a1c021b $ solana-ledger-tool --ledger /tmp/mainnet-beta-archive-removal/ verify genesis hash: 5eykt4UsFv8P8NJdTREpY1vzqKqZKvdpKuc147dw2N9d [2020-05-15T05:54:34.009676472Z INFO solana_ledger::blockstore] Maximum open file descriptors: 65000 [2020-05-15T05:54:34.009743474Z INFO solana_ledger::blockstore] Opening database at "/tmp/mainnet-beta-archive-removal/rocksdb" [2020-05-15T05:54:34.027076692Z INFO solana_ledger::blockstore] "/tmp/mainnet-beta-archive-removal/rocksdb" open took 17ms [2020-05-15T05:54:34.027164924Z INFO solana_ledger::bank_forks_utils] Initializing snapshot path: "/tmp/mainnet-beta-archive-removal/snapshot" [2020-05-15T05:54:34.032782425Z INFO solana_ledger::bank_forks_utils] Loading snapshot package: "/tmp/mainnet-beta-archive-removal/snapshot-11079842-G3twNav4jRDVv1nLZyVSd4nALMRXEDRf4g9Jqg8PgZNJ.tar.bz2" [2020-05-15T05:55:02.440804865Z INFO solana_ledger::snapshot_utils] snapshot untar took 28.4s [2020-05-15T05:55:02.440903305Z INFO solana_ledger::snapshot_utils] snapshot version: 1.1.0 [2020-05-15T05:55:02.440985031Z INFO solana_ledger::snapshot_utils] Loading bank from "/tmp/mainnet-beta-archive-removal/snapshot/.tmp0YB9CQ/snapshots/11079842/11079842" [2020-05-15T05:55:02.510126180Z INFO solana_ledger::snapshot_utils] Rebuilding accounts... [2020-05-15T05:57:40.921624347Z INFO solana_ledger::snapshot_utils] Rebuilding status cache... [2020-05-15T05:57:41.216539709Z INFO solana_ledger::snapshot_utils] Loaded bank for slot: 11079842 [2020-05-15T05:57:41.217804233Z INFO solana_runtime::accounts_db] total_stores: 2608, newest_slot: 11079842, oldest_slot: 0, max_slot: 0 (num=5), min_slot: 8257176 (num=1) [2020-05-15T05:57:41.217899331Z INFO solana_metrics::metrics] metrics disabled: SOLANA_METRICS_CONFIG: environment variable not found [2020-05-15T05:57:41.218171778Z INFO solana_metrics::metrics] datapoint: accounts_db-stores total_count=2608i [2020-05-15T05:57:41.218300031Z INFO solana_metrics::metrics] datapoint: accounts_db-perf-stats delta_hash_num=0i delta_hash_scan_us=0i delta_hash_merge_us=0i delta_hash_accumulate_us=0i [2020-05-15T05:57:43.956916251Z INFO solana_runtime::bank] bank frozen: 11079842 hash: 9Aj3XorHCeVk1JKiV1vKjBqVB8FRiJze3gPx2FqohT2g accounts_delta: EjR393U12AeojVPBVCAjAjgjhhz5Mix7ce4JgnsAB5Za signature_count: 61 last_blockhash: zru76SK5dQqeZQd9VEZt8bTzPWSi2MXSb5VZvc8QTNj capitalization: 499996862353074280 [2020-05-15T05:57:43.957176578Z INFO solana_runtime::bank] accounts hash slot: 11079842 stats: BankHashStats { num_updated_accounts: 128, num_removed_accounts: 0, num_lamports_stored: 25024604877564, total_data_len: 385232, num_executable_accounts: 0 } [2020-05-15T05:57:43.957229243Z INFO solana_ledger::snapshot_utils] bank rebuild from snapshot took 161.5s [2020-05-15T05:57:43.959141280Z INFO solana_ledger::blockstore_processor] processing ledger from slot 11079842... [2020-05-15T05:57:43.976406426Z INFO solana_ledger::blockstore_processor] ledger holds data through slot 11081264 [2020-05-15T05:57:45.123082143Z INFO solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=11079843i block_height=10730179i [2020-05-15T05:57:48.930712400Z INFO solana_runtime::bank] bank frozen: 11079843 hash: BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz accounts_delta: A45j5o3dhZDmyTnXeFZ4RnDy2FV4Ge8hTgMQSE9eZ4So signature_count: 96 last_blockhash: CX98iQ5TxS7QH9dfTW9x6r1NSkBkAcKy9kJvJm8wKf1j capitalization: 499996862352594280 [2020-05-15T05:57:48.930933184Z INFO solana_runtime::bank] accounts hash slot: 11079843 stats: BankHashStats { num_updated_accounts: 198, num_removed_accounts: 0, num_lamports_stored: 37404817053384, total_data_len: 515817, num_executable_accounts: 0 } [2020-05-15T05:57:48.931963735Z INFO solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=11079844i block_height=10730180i [2020-05-15T05:57:48.941486692Z INFO solana_ledger::blockstore_processor] processing ledger: slot=11079844, last root slot=11079842 slots=1 slots/s=0.33333334 txs/s=32 [2020-05-15T05:57:52.426843011Z WARN solana_vote_program::vote_state] 6TkKqq15wXjqEjNg9zqTKADwuVATR9dW3rkNnsYme1ea dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:52.447795251Z WARN solana_vote_program::vote_state] 3B2mGaZoFwzAnWCoZ4EAKdps4FbYbDKQ48jo8u1XWynU dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:52.524839169Z WARN solana_vote_program::vote_state] E6cyDdEH8fiyCTusmWcZVhapAvvp2LK24zMLg4KrrAkt dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:52.671383611Z INFO solana_runtime::bank] bank frozen: 11079844 hash: AxeMqNouZYLXy63L7e7VvAEKmawYUCgrUfmCuoWFMmiQ accounts_delta: CeRrB6gjHc8tRCafiWXYmV1Z9F3iR6Tt1Fa89A5h99u9 signature_count: 85 last_blockhash: BeVeHQHvVZNPrHkbnw7iAMDysmTq42dQ88g44Rj1WRoQ capitalization: 499996862352169280 [2020-05-15T05:57:52.671607950Z INFO solana_runtime::bank] accounts hash slot: 11079844 stats: BankHashStats { num_updated_accounts: 169, num_removed_accounts: 0, num_lamports_stored: 34201276149524, total_data_len: 448659, num_executable_accounts: 0 } [2020-05-15T05:57:52.673137686Z INFO solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=11079845i block_height=10730181i [2020-05-15T05:57:52.682822768Z INFO solana_ledger::blockstore_processor] processing ledger: slot=11079845, last root slot=11079842 slots=1 slots/s=0.33333334 txs/s=28.333334 [2020-05-15T05:57:55.487834717Z WARN solana_vote_program::vote_state] 3nRhescC7HMYC8gKti3ENiBe8LnKZUs2gzYPAjYniQCP dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.495472594Z WARN solana_vote_program::vote_state] 9CqDvvGvSNVZDo5RskCAv4fTubpFCs9RLTrjUxEYrvNA dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.502889878Z WARN solana_vote_program::vote_state] 6kDyGMHbuWekkcquroYNp8VRL5pQiUzEJ11gJ75qJsRy dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.505894361Z WARN solana_vote_program::vote_state] DE1bawNcRJB9rVm3buyMVfr8mBEoyyu73NBovf2oXJsJ dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.508948632Z WARN solana_vote_program::vote_state] 3fA6TU7fQCkNDDKYJeCY4Ag2gCatEsYnYL4SpkSDYfCw dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.511854029Z WARN solana_vote_program::vote_state] A79u1awz7CqnxmNYEVtzWwSzup3eKPNW6w2Jrd56oZ3y dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.514803954Z WARN solana_vote_program::vote_state] cZCvgqgrdX2LCScVku8CprMnAUZtSicBDJ8eiowfonn dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.519254908Z WARN solana_vote_program::vote_state] 9gJT69qJUibNmfnAPBgsqSmLqVxc3kxXpw1Vk7APDrso dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.522454760Z WARN solana_vote_program::vote_state] 5XKJwdKB2Hs7pkEXzifAysjSk6q7Rt6k5KfHwmAMPtoQ dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.525366659Z WARN solana_vote_program::vote_state] 6cgsK8ph5tNUCiKG5WXLMZFX1CoL4jzuVouTPBwPC8fk dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.528209824Z WARN solana_vote_program::vote_state] uEhHSnCXvWgtgvVaYscPHjG13G3peMmngQQ2ghC54i3 dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.530964715Z WARN solana_vote_program::vote_state] 5ya8UPujuXN8cys1EaaqMMH3auby3HTHZki73Q4Yfkff dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.533687773Z WARN solana_vote_program::vote_state] ateamuvZX4iy2xYrNrMyGgwtTpWFiurNzHpmJMZwjar dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.536402743Z WARN solana_vote_program::vote_state] 4XspXDcJy3DWZsVdaXrt8pE1xhcLpXDKkhj9XyjmWWNy dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.539146376Z WARN solana_vote_program::vote_state] 6TkKqq15wXjqEjNg9zqTKADwuVATR9dW3rkNnsYme1ea dropped vote Vote { slots: [11079844], hash: 4oNAwZxvcHMbSeqRqoG9ra4LxRtvE2uo1e2yeHgjL4i9, timestamp: None } failed to match hash 4oNAwZxvcHMbSeqRqoG9ra4LxRtvE2uo1e2yeHgjL4i9 AxeMqNouZYLXy63L7e7VvAEKmawYUCgrUfmCuoWFMmiQ [2020-05-15T05:57:55.541903544Z WARN solana_vote_program::vote_state] 4tS3UZfuRHzXuPenvErtRPtnZfY1KHhT96JBCQsLzKqW dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.544619415Z WARN solana_vote_program::vote_state] 8gcMmaEAXfRZKcXHeHV2x8R1ebb78inTr9xhhEuNNoTt dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.547436315Z WARN solana_vote_program::vote_state] 4YGgmwyqztpJeAi3pzHQ4Gf9cWrMHCjZaWeWoCK6zz6X dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.550265502Z WARN solana_vote_program::vote_state] LunaowJnt875WWoqDkhHhE93SNYHa6tfFNVn1rqc57c dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.553003427Z WARN solana_vote_program::vote_state] CakcnaRDHka2gXyfbEd2d3xsvkJkqsLw2akB3zsN1D2S dropped vote Vote { slots: [11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.555700183Z WARN solana_vote_program::vote_state] Dq3piY2ZcBvNN84j2EhDLtTzRAw95za7Eau89pNcmSd5 dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.558433748Z WARN solana_vote_program::vote_state] 2dxz129YxB1xtf7Mx6HUT5JspexArNNtQt84FYueWZV7 dropped vote Vote { slots: [11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.561177872Z WARN solana_vote_program::vote_state] 3RXKQBRv7xKTQeNdLSPhCiD4QcUfxEQ12rtgUkMf5LnS dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.566828821Z WARN solana_vote_program::vote_state] MCFmmmXdzTKjBEoMggi8JGFJmd856uYSowuH2sCU5kx dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.577144772Z WARN solana_vote_program::vote_state] Fd7btgySsrjuo25CJCj7oE7VPMyezDhnx7pZkj2v69Nk dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.578795118Z WARN solana_vote_program::vote_state] CjmXSapt1ouz3CZzgkRJckBEwMSo5fVdVrizLeRscwYD dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.579891372Z WARN solana_vote_program::vote_state] ChorusmmK7i1AxXeiTtQgQZhQNiXYU84ULeaYF1EH15n dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.582553065Z WARN solana_vote_program::vote_state] DDnAqxJVFo2GVTujibHt5cjevHMSE9bo8HJaydHoshdp dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.585256122Z WARN solana_vote_program::vote_state] 452L4U6HbzT59EP9vLyPxddF99FBBZ7foCrcn5A9HLMK dropped vote Vote { slots: [11079842, 11079843], hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP, timestamp: None } failed to match hash 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz [2020-05-15T05:57:55.605849043Z INFO solana_runtime::bank] bank frozen: 11079845 hash: 679yiohYNK2ZYgyTxTMUAhXJtZdvTwoZbsvZrT2VmxzJ accounts_delta: 8r69C1rT6rNnPEiRmzv23Tvt86oDDbSY27v5M6wDLy9R signature_count: 33 last_blockhash: 4reo5wDHFe3nhtUXT33zXCVMwFs8F8nFZaEiCGxZuroL capitalization: 499996862352004280 [2020-05-15T05:57:55.606065384Z INFO solana_runtime::bank] accounts hash slot: 11079845 stats: BankHashStats { num_updated_accounts: 43, num_removed_accounts: 0, num_lamports_stored: 13456795155604, total_data_len: 172565, num_executable_accounts: 0 }

test result examination

When you're examining the preceding logs, you can notice the initial bank resumed from the snapshot results in the same bankhash for each cases:

# good [2020-05-15T05:41:54.166416330Z INFO solana_runtime::bank] bank frozen: 11079842 hash: 9Aj3XorHCeVk1JKiV1vKjBqVB8FRiJze3gPx2FqohT2g accounts_delta: EjR393U12AeojVPBVCAjAjgjhhz5Mix7ce4JgnsAB5Za signature_count: 61 last_blockhash: zru76SK5dQqeZQd9VEZt8bTzPWSi2MXSb5VZvc8QTNj capitalization: 499996862353074280 # bad [2020-05-15T05:57:43.956916251Z INFO solana_runtime::bank] bank frozen: 11079842 hash: 9Aj3XorHCeVk1JKiV1vKjBqVB8FRiJze3gPx2FqohT2g accounts_delta: EjR393U12AeojVPBVCAjAjgjhhz5Mix7ce4JgnsAB5Za signature_count: 61 last_blockhash: zru76SK5dQqeZQd9VEZt8bTzPWSi2MXSb5VZvc8QTNj capitalization: 499996862353074280

But the immediate child bank's hash changed:

# good [2020-05-15T05:42:00.169123981Z INFO solana_runtime::bank] bank frozen: 11079843 hash: 79FHJykGmzLTECqCaKwCjAjZVb569fdGbAQ6pQ9xnaoP accounts_delta: 9gAMaX1o9gzK2sCSWVsg9PHtH8HrsDAhjNW1XDu2omcx signature_count: 96 last_blockhash: CX98iQ5TxS7QH9dfTW9x6r1NSkBkAcKy9kJvJm8wKf1j capitalization: 499996862352594280 # bad [2020-05-15T05:57:48.930712400Z INFO solana_runtime::bank] bank frozen: 11079843 hash: BmfTfqU3RLDKwiUezbx4zZzjBsQktZLkQrSbh6iM2sLz accounts_delta: A45j5o3dhZDmyTnXeFZ4RnDy2FV4Ge8hTgMQSE9eZ4So signature_count: 96 last_blockhash: CX98iQ5TxS7QH9dfTW9x6r1NSkBkAcKy9kJvJm8wKf1j capitalization: 499996862352594280

I highly suspect this is due to segment always being 0.

FYI: I've dumped bank.clock() like this and got that:

diff --git a/ledger/src/snapshot_utils.rs b/ledger/src/snapshot_utils.rs index 1c8b8b9a3..4d63af255 100644 --- a/ledger/src/snapshot_utils.rs +++ b/ledger/src/snapshot_utils.rs @@ -647,6 +647,7 @@ where } }; bank.operating_mode = Some(genesis_config.operating_mode); + error!("ryoqun bank.clock(): {:?}", bank.clock()); info!("Rebuilding accounts..."); let rc = bank::BankRc::from_stream( account_paths,

# good [2020-05-15T03:36:28.239419844Z ERROR solana_ledger::snapshot_utils] ryoqun bank.clock(): Clock { slot: 11070792, segment: 10812, epoch: 25, leader_schedule_epoch: 26, unix_timestamp: 1588797256 } # bad [2020-05-15T04:50:36.024245871Z ERROR solana_ledger::snapshot_utils] ryoqun bank.clock(): Clock { slot: 11079842, unused: 0, epoch: 25, leader_schedule_epoch: 26, unix_timestamp: 1588800876 }

how to prepare recent snapshot and ledger

First we need to fetch snapshot and recent slot shreds to replay by running ad-hoc validator like this:

$ mkdir /tmp/mainnet-beta-archive-removal $ solana-validator --ledger /tmp/mainnet-beta-archive-removal/ --entrypoint mainnet-beta.solana.com:8001 --log - --private-rpc --rpc-port 8665 --snapshot-interval-slots 0 --dynamic-port-range 12000-12010 --trusted-validator 7Np41oeYqPefeNQEHSv1UDhYrehxin3NStELsSKCT4K2 --trusted-validator GdnSyH3YtwcxFvQrVVJMm1JhTS4QVX7MFsX56uJLUfiZ --trusted-validator DE1bawNcRJB9rVm3buyMVfr8mBEoyyu73NBovf2oXJsJ --trusted-validator CakcnaRDHka2gXyfbEd2d3xsvkJkqsLw2akB3zsN1D2S --no-untrusted-rpc --expected-shred-version 54208 --expected-genesis-hash 5eykt4UsFv8P8NJdTREpY1vzqKqZKvdpKuc147dw2N9d

Run it until it says bunch of bank frozen messages like this:

[2020-05-15T04:56:30.108808174Z INFO solana_runtime::bank] bank frozen: 11079845 hash: 679yiohYNK2ZYgyTxTMUAhXJtZdvTwoZbsvZrT2VmxzJ accounts_delta: 8r69C1rT6rNnPEiRmzv23Tvt86oDDbSY27v5M6wDLy9R signature_count: 33 last_blockhash: 4reo5wDHFe3nhtUXT33zXCVMwFs8F8nFZaEiCGxZuroL capitalization: 499996862352004280

Then Ctrl-C!

After that, we can avoid to run an adhoc validator to check the re-playability of the ledger like this:

$ solana-validator --ledger /tmp/mainnet-beta-archive-removal/ --verify

At this stage, the required data to run ledger-tool verify is stored at that directory. So there is no need to run validator each time. ledger-tool is way more easy to handle and be reproducible.

Thanks @ryoqun for the awesome writeup. I was able to replicate the failure, this PR should address it: #10078

ryoqun · 2020-05-14T02:35:12Z

runtime/src/bank.rs

@@ -237,6 +235,19 @@ impl HashAgeKind {
    }
 }

+#[derive(Default, Clone, PartialEq, Debug, Deserialize, Serialize)]
+struct UnusedAccounts {


nice placeholder! (I didn't notice this way did exist to be compatible with that easy cost.)

Pull request has been modified.

automerge

ryoqun · 2020-07-20T07:23:19Z

runtime/src/accounts.rs

-                            && !(account.lamports == std::u64::MAX
-                                && account.owner == solana_storage_program::id())


Similar handling with this is removed from ledger-tool capitalization at #11107

jackcmay commented May 12, 2020

View reviewed changes

runtime/src/bank.rs Show resolved Hide resolved

jackcmay commented May 13, 2020

View reviewed changes

cli/src/cli.rs Outdated Show resolved Hide resolved

jackcmay force-pushed the rip-out-storage branch from 7457fcd to c96f6f7 Compare May 13, 2020 18:14

jackcmay requested review from sakridge and mvines May 13, 2020 18:16

jackcmay commented May 13, 2020

View reviewed changes

danpaul000 previously approved these changes May 13, 2020

View reviewed changes

sakridge previously approved these changes May 13, 2020

View reviewed changes

jackcmay commented May 14, 2020

View reviewed changes

ryoqun reviewed May 14, 2020

View reviewed changes

runtime/src/bank.rs Show resolved Hide resolved

ryoqun reviewed May 14, 2020

View reviewed changes

Remove archiver and storage program

d65962b

jackcmay force-pushed the rip-out-storage branch from b0f23c5 to d65962b Compare May 14, 2020 23:35

jackcmay added the automerge Merge this Pull Request automatically once CI passes label May 15, 2020

solana-grimes merged commit eb1acaf into solana-labs:master May 15, 2020

ryoqun mentioned this pull request May 15, 2020

9951 clippy errors in the test suite #10030

Merged

jackcmay deleted the rip-out-storage branch May 15, 2020 03:18

ryoqun mentioned this pull request May 16, 2020

fix clock bankhash mismatch #10078

Merged

mvines mentioned this pull request May 17, 2020

Deal with misleading storage rewards in Inflation struct #10092

Closed

ryoqun mentioned this pull request May 18, 2020

sdk-c requires separate cargo clippy run on CI #5503

Closed

sakridge linked an issue May 20, 2020 that may be closed by this pull request

Function process_turn() only used in CUDA #9928

Closed

sakridge mentioned this pull request May 20, 2020

Function process_turn() only used in CUDA #9928

Closed

danpaul000 pushed a commit to danpaul000/solana that referenced this pull request Jul 8, 2020

Remove archiver and storage program (solana-labs#9992)

fce7ccb

automerge

ryoqun reviewed Jul 20, 2020

View reviewed changes

ryoqun mentioned this pull request Sep 1, 2020

Check bank capitalization #11927

Merged

casrhub mentioned this pull request Aug 26, 2024

Archivers — Distributed ledger store documentation missing solana-foundation/developer-content#350

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove archiver and storage program #9992

Remove archiver and storage program #9992

jackcmay commented May 11, 2020

jackcmay commented May 11, 2020

codecov bot commented May 12, 2020 •

edited

Loading

sakridge commented May 12, 2020 •

edited

Loading

sakridge commented May 12, 2020

jackcmay commented May 12, 2020

sakridge commented May 12, 2020

jackcmay commented May 13, 2020

carllin commented May 13, 2020

jackcmay commented May 13, 2020

carllin commented May 13, 2020

jackcmay May 13, 2020

carllin May 13, 2020

jackcmay commented May 13, 2020

danpaul000 left a comment

sakridge left a comment

jackcmay May 14, 2020

sakridge May 14, 2020

ryoqun May 14, 2020

jackcmay May 14, 2020

ryoqun May 15, 2020

jackcmay May 15, 2020

ryoqun May 15, 2020 •

edited

Loading

ryoqun May 15, 2020

ryoqun May 15, 2020

jackcmay May 15, 2020 •

edited

Loading

ryoqun May 14, 2020 •

edited

Loading

ryoqun Jul 20, 2020

		&& !(account.lamports == std::u64::MAX
		&& account.owner == solana_storage_program::id())

Remove archiver and storage program #9992

Remove archiver and storage program #9992

Conversation

jackcmay commented May 11, 2020

Problem

Summary of Changes

jackcmay commented May 11, 2020

codecov bot commented May 12, 2020 • edited Loading

Codecov Report

sakridge commented May 12, 2020 • edited Loading

sakridge commented May 12, 2020

jackcmay commented May 12, 2020

sakridge commented May 12, 2020

jackcmay commented May 13, 2020

carllin commented May 13, 2020

jackcmay commented May 13, 2020

carllin commented May 13, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jackcmay commented May 13, 2020

danpaul000 left a comment

Choose a reason for hiding this comment

sakridge left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryoqun May 15, 2020 • edited Loading

Choose a reason for hiding this comment

TEST SETUP

TEST STEP

Good (at a914bc5b7)

Bad (at 9a1c021ba)

Choose a reason for hiding this comment

test result examination

Choose a reason for hiding this comment

how to prepare recent snapshot and ledger

jackcmay May 15, 2020 • edited Loading

Choose a reason for hiding this comment

ryoqun May 14, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented May 12, 2020 •

edited

Loading

sakridge commented May 12, 2020 •

edited

Loading

ryoqun May 15, 2020 •

edited

Loading

Good (at `a914bc5b7`)

Bad (at `9a1c021ba`)

jackcmay May 15, 2020 •

edited

Loading

ryoqun May 14, 2020 •

edited

Loading