Wire unified scheduler into banking experimentally

[ryoqun]

Problem

Currently, the unified scheduler can't be used as a block-production method.

Summary of Changes

(See Proposition/justification section at #3832 for justification of this pr in the first place)

This pr (and any spin-off ones) implements all the necessary plumbing from the cli to the innermost (SchedulingStateMachine) of the unified scheduler with minimal functionality. Note that, even after all the changes in this pr is merged, unified scheduler isn't fully functional nor performant nor secure for the banking stage in any sense.

as part of the plumbing, new thread management mechanism is introduced inside unified-scheduler-pool, parallel to the already-existing one for the unified scheduler in the block verification. As a quick recap, unified scheduler assigns independent thread pools to each active banks for ongoing forks while verifying blocks.

on the other hand, unified scheduler maintains one thread pool at most (as a singleton object) for the current working_bank of PohRecorder, when producing blocks. This means these two kinds of thread pools (one for block verification, other for block production) are managed differently. Other than that, most of core code (thread looping and SchedulingStateMachine) is still shared among them with limited number of branching (i.e. I'd say it's still unified lol).

This differentiated behavior at the higher layer is intentional to accommodate banking-stage specific requirements:

(1) existing relevant code base is strongly assuming non-concurrent block production even if there are multiple competing forks, which i think a justifiable design decision for impl simplicity. This means the channel receivers (nonvote_receiver, tpu_receiver, gossip_receiver) for incoming transactions are kind of a singleton set of resources and aren't fork-aware. Also, unified scheduler needs some preprocessing (SanitizedTransaction/TransactioView creation, alt resolution, and Pubkey->UsageQueue lookups). With all these considerations in mind, the channels are directly connected to unified-scheduler's handler thread main loop of a particular scheduler instance, in order to do the offloaded processing there multi-threadedly, while avoiding extra messaging pop for efficiency.

the actual integration are implemented with above-mentioned design goals. at validator startup, if BankingStage::new() is branched into new_unified_scheduler(), it does minimal setup even without spawning threads. Specifically, it just calls scheduler_pool.register_banking_stage(). its notable arg is the callback closure to convert BankingPacketBatches to SanitizedTransactions, which is actually run on handler threads.
The closure type is slightly involved because of the need to respawn thread pool from time to time to mitigate unbounded UsageQueueLoader growth. So, the closure is nested: the inner closure for convention is called BatchConverter. The outer closure to create the converter is called BatchConverterCreator. Along with the closure, the actual transaction-incoming channel and small monitor object (called BankingStageMonitor) are registered to the unified scheduler pool. These args are retained as-is, typed as BlockProductionSchedulerRespawner, internally. These separation incurs a single dynamic dispatch, but intentionally done for separation of concerns.

(2) transaction buffering (and prioritization-ordering) is desired to be run prior to the first bank creation and this buffer must be carried over to the first bank of leader slots and the next, etc. So, block-verifying unified scheduler's tight coupling of 1 Bank-to-1 thread pool isn't appropriate here.

towards that end, now thread pool can be started without a bank by making SchedulingContext::bank Option<_>-ed. Then, it enters into the buffering state immediately after the setup. Previously, this wasn't possible.

Also, scheduling session needed some adjustment to reach to safe suspension point quickly, so that next child bank could resume timely. As a quick recap, SchedulingStateMachine didn't previously buffer any non-conflicting ready-to-execute transactions. That meant these transactions are buffered inside crossbeam channels. And, its (potentially large) buffer has to be cleared before switching to the next bank, to maintain runtime invariant.

So, SchedulingStateMachine now starts to buffer transaction internally up-to fixed cap instead. And scheduler thread's main loop introduces new state for session, to be paired to the new buffering mechanism: session_pausing = true

Note that, this scheduling resumption mechanism doesn't work nicely when switching forks in the middle of leader slots. Specifically, it doesn't recover already-processed higher-paying txes, which are yet to be executed at the new fork. This behavior is also true for central scheduler.

As a bonus, session_resetting = true is also introduced to clear any remaining unprocessed buffered transactions after leader schedule.

The actual change list:

1. preparations

• Introduce BankingTracer::create_channels(), to use same channels over all of {nonvote,tpu,gossip}_channels. Define BankingTracer::create_channels() #4041

• Promote receiver payload types solana-core to solana-perf (SigverifyTracerPacketStats (EDIT: obsoleted, thanks to Remove Tracer packet as a concept #4043) /BankingPacketBatch/BankingPacketReceiver). Apply cleanups to solana-core for unified scheduler #4123

  • This is needed to directly use BankingPacketBatch/BankingPacketReceiver inside solana-unified-scheduler-pool.

• Support AbiExample for std::num::Saturating<T>.

  • This is needed because SigverifyTracerPacketStats used solana_sdk's saturating_add macro previously. Now solana_perf doesn't depend on solana-sdk. (EDIT: obsoleted, thanks to Remove Tracer packet as a concept #4043)

• Make TransactionHandler trait's api more general: Make TaskHandler::handle() more generalized #4050

  • this is needed for additional committing to poh recorder.

• Move RootBankCache creation to avoid deadlock: Apply cleanups to solana-core for unified scheduler #4123

• Define SchedulerInner, a private trait for internal use by solana-unified-scheuler-pool: Define SchedulerInner for fine-grained cleaning #4133
  commit between solana-unified-scheduler, solana-runtime, solana-ledger.

  • Adjust load_execute_and_commit_transactions api as well for this.

• Introduce TransactionError::CommitFailed to propagate poh faiture immediately before bank: Support tx poh recording in unified scheduler #4150
  commit between solana-unified-scheduler-pool, solana-runtime, solana-ledger.

  • Adjust load_execute_and_commit_transactions api as well for this.

• To beatify the upcoming diffs: Apply misc small cleanups to unified scheduler #4080 Apply more cosmetic changes to unified scheduler #4111

2. actual plumbing

• Introduce SchedulingMode::{BlockVerification, BlockProduction} finally!
  • Make BankForks/InstalledScheduler aware of SchedulingMode.
• Make SchedulingStateMachine buffer ready-to-execute tasks internally up to max_executing_task
• Selectively gate the cli value of unified-scheduler for --block-production-method, with override flag (--enable-experimental-block-production-method)
• Make PohRecorder::reset() return overwritten BankWithScheduler to reap now-unused scheduler from the bank wrapper.
• Add minimal local-cluster test: test_randomly_mixed_block_production_methods_between_bootstrap_and_not
• Make agave-ledger-tool simulate-block-production support --block-production-method unified-scheduler
• Make solana-banking-bench simulate-block-production support --block-production-method unified-scheduler

Note that this pr is extracted from #2325

[apfitzge]

nothing new in sdk, please 🙏

[ryoqun]

okay, you spotted that. :) 723f765

[ryoqun]

this (backref: #4123 (comment))

[apfitzge]

a concern here is that elsewhere we grab these locks in the opposite order.
I'm fairly certain we don't but given these are both so extensively used (smelly) I'm not convinced entirely that we do not.

Eventually there should be some other delivery system for TPU banks, other than locking poh. I'm not quite sure what that should look like though. Any thoughts?

[ryoqun]

this (backref: #4123 (comment))

[apfitzge]

See the above comment on the lock order - feel it may be worth investing in a better way to deliver working banks to tpu to avoid these weird edge cases where we need to be very careful.

[ryoqun]

it seems that that wall of text couldn't help and i crossed your code-smell threshold nevertheless.. ;)

how about this? 110c1bc (note: ci won't pass)

if it's okay for you that unified scheduler code breaks out of the BankingStage code organization more aggressively like that, i can remove this weird edge case: 9839ab8

feel it may be worth investing in a better way to deliver working banks to tpu to avoid these weird edge cases where we need to be very careful.

fyi, this is completely different from the update_bank_forks_and_poh_recorder_for_new_tpu_bank thing (tpu delivery as you said), which i'll plan to address later with a different commit.