move maybe_evict_cache() off default rayon pool

[msmouse]

Description

This can potentially block the pre_commit_ledger thread while it's holding the buffered_state lock. Before we reduce the scope of that lock, I'm moving this part off the defualt pool so that even if the default pool gets busy (or worse, there are tasks on the threads of the pool trying to lock the buffered_state) this part doesn't get affected.

How Has This Been Tested?

Key Areas to Review

Type of Change

• New feature

Which Components or Systems Does This Change Impact?

• Validator Node

[trunk-io]

⏱️ 2h 31m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
rust-smoke-tests	1h 27m	🟥 🟥 🟩
execution-performance / single-node-performance	21m	🟩
test-target-determinator	8m	🟩 🟩
rust-cargo-deny	5m	🟩 🟩 🟩
check-dynamic-deps	5m	🟩 🟩 🟩 🟩
rust-doc-tests	5m	🟩
execution-performance / test-target-determinator	4m	🟩
check	4m	🟩
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
semgrep/ci	2m	🟩 🟩 🟩 🟩
rust-move-tests	2m	🟩
fetch-last-released-docker-image-tag	1m	🟩
general-lints	1m	🟩 🟩 🟩
file_change_determinator	32s	🟩 🟩 ⬜ 🟩

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[github-actions]

✅ Forge suite realistic_env_max_load success on a29d23639745500ecebd263ade708544100c962b

two traffics test: inner traffic : committed: 13917.57 txn/s, latency: 2853.20 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 3300 ms), latency samples: 5291780
two traffics test : committed: 100.00 txn/s, latency: 1495.74 ms, (p50: 1500 ms, p70: 1500, p90: 1600 ms, p99: 1800 ms), latency samples: 1800
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.060, avg: 1.619", "ConsensusProposalToOrdered: max: 0.341, avg: 0.303", "ConsensusOrderedToCommit: max: 0.382, avg: 0.370", "ConsensusProposalToCommit: max: 0.681, avg: 0.673"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.10s no progress at version 2071742 (avg 0.21s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.20s no progress at version 2071740 (avg 6.89s) [limit 15].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite framework_upgrade success on b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> a29d23639745500ecebd263ade708544100c962b

Compatibility test results for b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> a29d23639745500ecebd263ade708544100c962b (PR)
Upgrade the nodes to version: a29d23639745500ecebd263ade708544100c962b
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1077.62 txn/s, submitted: 1080.15 txn/s, failed submission: 2.53 txn/s, expired: 2.53 txn/s, latency: 2873.92 ms, (p50: 2600 ms, p70: 2900, p90: 4500 ms, p99: 5900 ms), latency samples: 93800
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1131.73 txn/s, submitted: 1134.17 txn/s, failed submission: 2.44 txn/s, expired: 2.44 txn/s, latency: 2660.22 ms, (p50: 2500 ms, p70: 3000, p90: 3900 ms, p99: 5400 ms), latency samples: 101920
5. check swarm health
Compatibility test for b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> a29d23639745500ecebd263ade708544100c962b passed
Upgrade the remaining nodes to version: a29d23639745500ecebd263ade708544100c962b
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1182.88 txn/s, submitted: 1184.86 txn/s, failed submission: 1.98 txn/s, expired: 1.98 txn/s, latency: 2512.53 ms, (p50: 2400 ms, p70: 2700, p90: 3900 ms, p99: 6300 ms), latency samples: 107360
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite compat success on b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> a29d23639745500ecebd263ade708544100c962b

Compatibility test results for b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> a29d23639745500ecebd263ade708544100c962b (PR)
1. Check liveness of validators at old version: b29f09f57e898d8d211c8bc3e303f6e50bba2266
compatibility::simple-validator-upgrade::liveness-check : committed: 17199.17 txn/s, latency: 1973.88 ms, (p50: 2000 ms, p70: 2100, p90: 2200 ms, p99: 2400 ms), latency samples: 553120
2. Upgrading first Validator to new version: a29d23639745500ecebd263ade708544100c962b
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 5720.50 txn/s, latency: 5007.97 ms, (p50: 5300 ms, p70: 5700, p90: 6500 ms, p99: 6800 ms), latency samples: 114280
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 5859.37 txn/s, latency: 5533.77 ms, (p50: 5900 ms, p70: 6100, p90: 7300 ms, p99: 7500 ms), latency samples: 202560
3. Upgrading rest of first batch to new version: a29d23639745500ecebd263ade708544100c962b
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 5885.90 txn/s, latency: 4865.80 ms, (p50: 5600 ms, p70: 5800, p90: 6000 ms, p99: 6100 ms), latency samples: 112040
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 5512.13 txn/s, latency: 5828.73 ms, (p50: 6000 ms, p70: 6100, p90: 7600 ms, p99: 7900 ms), latency samples: 191660
4. upgrading second batch to new version: a29d23639745500ecebd263ade708544100c962b
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 8246.54 txn/s, latency: 3433.18 ms, (p50: 3800 ms, p70: 4000, p90: 4200 ms, p99: 4300 ms), latency samples: 153060
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 8512.81 txn/s, latency: 3757.82 ms, (p50: 3900 ms, p70: 4100, p90: 4400 ms, p99: 4600 ms), latency samples: 282800
5. check swarm health
Compatibility test for b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> a29d23639745500ecebd263ade708544100c962b passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking