[cp][aptos-release-v1.20] Fix potential deadlock in state_key::Entry::drop

[github-actions]

Backport

This will backport the following commits from main to aptos-release-v1.20:

• Fix potential deadlock in state_key::Entry::drop (#14670)

Questions ?

Please refer to the Backport tool documentation

[trunk-io]

⏱️ 2h 6m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
execution-performance / single-node-performance	23m	🟩
rust-unit-tests	22m	🟩
forge-compat-test / forge	19m	🟩
forge-framework-upgrade-test / forge	19m	🟩
forge-e2e-test / forge	12m	🟩
rust-move-tests	9m	🟩
rust-doc-tests	5m	🟩
test-target-determinator	4m	🟩
execution-performance / test-target-determinator	4m	🟩
check	4m	🟩
general-lints	2m	🟩
rust-cargo-deny	2m	🟩
check-dynamic-deps	36s	🟩
semgrep/ci	19s	🟩
file_change_determinator	12s	🟩
file_change_determinator	11s	🟩
file_change_determinator	9s	🟩
permission-check	3s	🟩
Backport PR	3s	🟥
permission-check	3s	🟩
permission-check	2s	🟩
permission-check	2s	🟩
determine-docker-build-metadata	2s	🟩
permission-check	2s	🟩
permission-check	2s	🟩

🚨 1 job on the last run was significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
execution-performance / single-node-performance	23m	18m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[github-actions]

✅ Forge suite realistic_env_max_load success on 30d8e3783cea9ff4250d8bbc24cbd3daa1f1a662

two traffics test: inner traffic : committed: 14676.54 txn/s, latency: 2706.92 ms, (p50: 2700 ms, p70: 2700, p90: 2900 ms, p99: 3200 ms), latency samples: 5580440
two traffics test : committed: 99.95 txn/s, latency: 1699.63 ms, (p50: 1500 ms, p70: 1600, p90: 1800 ms, p99: 10100 ms), latency samples: 1780
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.233, avg: 0.219", "QsPosToProposal: max: 1.048, avg: 1.011", "ConsensusProposalToOrdered: max: 0.311, avg: 0.287", "ConsensusOrderedToCommit: max: 0.414, avg: 0.405", "ConsensusProposalToCommit: max: 0.701, avg: 0.692"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.05s no progress at version 3133458 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 9.23s no progress at version 3133456 (avg 9.23s) [limit 15].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite framework_upgrade success on 70806ff543496aa6de7807feff49e7e1370efd20 ==> 30d8e3783cea9ff4250d8bbc24cbd3daa1f1a662

Compatibility test results for 70806ff543496aa6de7807feff49e7e1370efd20 ==> 30d8e3783cea9ff4250d8bbc24cbd3daa1f1a662 (PR)
Upgrade the nodes to version: 30d8e3783cea9ff4250d8bbc24cbd3daa1f1a662
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1228.48 txn/s, submitted: 1230.77 txn/s, failed submission: 2.28 txn/s, expired: 2.28 txn/s, latency: 2622.60 ms, (p50: 2400 ms, p70: 2700, p90: 4200 ms, p99: 5100 ms), latency samples: 107620
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1129.89 txn/s, submitted: 1131.91 txn/s, failed submission: 2.02 txn/s, expired: 2.02 txn/s, latency: 2792.40 ms, (p50: 2400 ms, p70: 3000, p90: 4500 ms, p99: 6000 ms), latency samples: 100880
5. check swarm health
Compatibility test for 70806ff543496aa6de7807feff49e7e1370efd20 ==> 30d8e3783cea9ff4250d8bbc24cbd3daa1f1a662 passed
Upgrade the remaining nodes to version: 30d8e3783cea9ff4250d8bbc24cbd3daa1f1a662
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1204.56 txn/s, submitted: 1205.97 txn/s, failed submission: 1.41 txn/s, expired: 1.41 txn/s, latency: 2678.39 ms, (p50: 2400 ms, p70: 3000, p90: 4100 ms, p99: 6200 ms), latency samples: 102460
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite compat success on 70806ff543496aa6de7807feff49e7e1370efd20 ==> 30d8e3783cea9ff4250d8bbc24cbd3daa1f1a662

Compatibility test results for 70806ff543496aa6de7807feff49e7e1370efd20 ==> 30d8e3783cea9ff4250d8bbc24cbd3daa1f1a662 (PR)
1. Check liveness of validators at old version: 70806ff543496aa6de7807feff49e7e1370efd20
compatibility::simple-validator-upgrade::liveness-check : committed: 16180.38 txn/s, latency: 2096.71 ms, (p50: 2100 ms, p70: 2100, p90: 2500 ms, p99: 3500 ms), latency samples: 525000
2. Upgrading first Validator to new version: 30d8e3783cea9ff4250d8bbc24cbd3daa1f1a662
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 6028.86 txn/s, latency: 4819.20 ms, (p50: 5200 ms, p70: 5600, p90: 6900 ms, p99: 7200 ms), latency samples: 126240
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 5890.98 txn/s, latency: 5437.38 ms, (p50: 5400 ms, p70: 5900, p90: 6700 ms, p99: 6900 ms), latency samples: 192860
3. Upgrading rest of first batch to new version: 30d8e3783cea9ff4250d8bbc24cbd3daa1f1a662
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7092.92 txn/s, latency: 3933.54 ms, (p50: 4200 ms, p70: 4400, p90: 5200 ms, p99: 5400 ms), latency samples: 139660
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7173.75 txn/s, latency: 4441.48 ms, (p50: 4500 ms, p70: 4700, p90: 6600 ms, p99: 6800 ms), latency samples: 244220
4. upgrading second batch to new version: 30d8e3783cea9ff4250d8bbc24cbd3daa1f1a662
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 11781.50 txn/s, latency: 2290.77 ms, (p50: 2300 ms, p70: 2700, p90: 3000 ms, p99: 3300 ms), latency samples: 205220
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10123.48 txn/s, latency: 2942.08 ms, (p50: 2500 ms, p70: 2800, p90: 3200 ms, p99: 8600 ms), latency samples: 397040
5. check swarm health
Compatibility test for 70806ff543496aa6de7807feff49e7e1370efd20 ==> 30d8e3783cea9ff4250d8bbc24cbd3daa1f1a662 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking