Replay stage panicked on mainnet-beta API node

[t-nelson]

Problem

thread 'solana-replay-stage' panicked at 'called `Option::unwrap()` on a `None` value', runtime/src/accounts_db.rs:857:29
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at ./cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/backtrace-0.3.44/src/backtrace/libunwind.rs:86

  ...

  15: solana_runtime::accounts_db::AccountsDB::clean_accounts
  16: solana_ledger::snapshot_utils::add_snapshot
  17: solana_ledger::bank_forks::BankForks::generate_snapshot
  18: solana_ledger::bank_forks::BankForks::set_root
  19: solana_core::replay_stage::ReplayStage::new::{{closure}}
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

log: https://drive.google.com/file/d/1zkxKc6QDoKm_myODIC03BS-Y6q20_cM6/view?usp=sharing

This node was running the v1.1.22 release

Proposed Solution

Triage and resolve

tag: @ryoqun @sakridge

[ryoqun]

@sakridge Hmm, it looks like the shrink code is very suspicious indeed...

[ryoqun]

Embarrassingly, since the very start of slot shrinking (#9219), it's been always racy with clean_accounts()....

I'm now writing a stress test to expose this.

[ryoqun]

I'm now writing a stress test to expose this.

status update: Wrote one and fixed the code. Pending on reviews. (FYI: @mvines: I'm marking this as a semi v1.2 upgrade blocker because I think we're better of fixing this security bug across testnet/mainnet consistently before jumping branches...)

[ryoqun]

I've confirmed this can occur on mainnet-beta and testnet very infrequently.

[ryoqun]

Useful queries for chronograf

SELECT time, host_id,program,thread, message FROM "mainnet-beta"."autogen"."panic" ORDER BY time DESC  

SELECT time, host_id,program,thread, message FROM "tds"."autogen"."panic" ORDER BY time DESC