[replay] Add error messages from aux data

[georgemitenkov]

Description

We should log auxiliary data when there is a transaction output mismatch to be able to see error messages in case those got removed.

How Has This Been Tested?

Key Areas to Review

Type of Change

• New feature
• Bug fix
• Breaking change
• Performance improvement
• Refactoring
• Dependency update
• Documentation update
• Tests

Which Components or Systems Does This Change Impact?

• Validator Node
• Full Node (API, Indexer, etc.)
• Move/Aptos Virtual Machine
• Aptos Framework
• Aptos CLI/SDK
• Developer Infrastructure
• Move Compiler
• Other (specify)

Checklist

• I have read and followed the CONTRIBUTING doc
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I identified and added all stakeholders and component owners affected by this change as reviewers
• I tested both happy and unhappy path of the functionality
• I have made corresponding changes to the documentation

[trunk-io]

⏱️ 1h 35m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
execution-performance / single-node-performance	48m	🟥 🟩
rust-move-tests	10m	🟩
rust-move-tests	9m	🟩
rust-doc-tests	5m	🟩
test-target-determinator	5m	🟩
execution-performance / test-target-determinator	4m	🟩
check	4m	🟩
rust-cargo-deny	3m	🟩 🟩
check-dynamic-deps	3m	🟩 🟩
fetch-last-released-docker-image-tag	2m	🟩
general-lints	58s	🟩 🟩
semgrep/ci	41s	🟩 🟩
file_change_determinator	23s	🟩 🟩
file_change_determinator	14s	🟩
Backport PR	6s	🟥

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[georgemitenkov]

• [replay] Add error messages from aux data #15156 👈
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @georgemitenkov and the rest of your teammates on Graphite

[github-actions]

✅ Forge suite realistic_env_max_load success on e034c3de390f41cb22b77d4b09d79de0f8c5a7bd

two traffics test: inner traffic : committed: 14261.76 txn/s, latency: 2784.63 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 3300 ms), latency samples: 5422680
two traffics test : committed: 100.06 txn/s, latency: 1480.37 ms, (p50: 1400 ms, p70: 1400, p90: 1600 ms, p99: 10200 ms), latency samples: 1840
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.014, avg: 1.592", "ConsensusProposalToOrdered: max: 0.323, avg: 0.294", "ConsensusOrderedToCommit: max: 0.372, avg: 0.359", "ConsensusProposalToCommit: max: 0.663, avg: 0.653"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.85s no progress at version 2808526 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.80s no progress at version 2808524 (avg 8.80s) [limit 15].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite framework_upgrade success on 1086a5e00d773704731ab84fb4ed3538613b2250 ==> e034c3de390f41cb22b77d4b09d79de0f8c5a7bd

Compatibility test results for 1086a5e00d773704731ab84fb4ed3538613b2250 ==> e034c3de390f41cb22b77d4b09d79de0f8c5a7bd (PR)
Upgrade the nodes to version: e034c3de390f41cb22b77d4b09d79de0f8c5a7bd
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1392.57 txn/s, submitted: 1394.91 txn/s, failed submission: 2.33 txn/s, expired: 2.33 txn/s, latency: 2430.66 ms, (p50: 2100 ms, p70: 2400, p90: 3700 ms, p99: 5400 ms), latency samples: 119340
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1235.66 txn/s, submitted: 1237.49 txn/s, failed submission: 1.84 txn/s, expired: 1.84 txn/s, latency: 2481.07 ms, (p50: 2100 ms, p70: 2400, p90: 4200 ms, p99: 6000 ms), latency samples: 107620
5. check swarm health
Compatibility test for 1086a5e00d773704731ab84fb4ed3538613b2250 ==> e034c3de390f41cb22b77d4b09d79de0f8c5a7bd passed
Upgrade the remaining nodes to version: e034c3de390f41cb22b77d4b09d79de0f8c5a7bd
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1243.79 txn/s, submitted: 1246.44 txn/s, failed submission: 2.65 txn/s, expired: 2.65 txn/s, latency: 2393.28 ms, (p50: 2300 ms, p70: 2600, p90: 3600 ms, p99: 5300 ms), latency samples: 112600
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite compat success on 1086a5e00d773704731ab84fb4ed3538613b2250 ==> e034c3de390f41cb22b77d4b09d79de0f8c5a7bd

Compatibility test results for 1086a5e00d773704731ab84fb4ed3538613b2250 ==> e034c3de390f41cb22b77d4b09d79de0f8c5a7bd (PR)
1. Check liveness of validators at old version: 1086a5e00d773704731ab84fb4ed3538613b2250
compatibility::simple-validator-upgrade::liveness-check : committed: 14532.97 txn/s, latency: 1975.14 ms, (p50: 1800 ms, p70: 1900, p90: 2400 ms, p99: 4900 ms), latency samples: 561840
2. Upgrading first Validator to new version: e034c3de390f41cb22b77d4b09d79de0f8c5a7bd
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 6007.52 txn/s, latency: 4874.00 ms, (p50: 5500 ms, p70: 5900, p90: 6100 ms, p99: 6200 ms), latency samples: 112260
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 5633.15 txn/s, latency: 5380.55 ms, (p50: 5800 ms, p70: 5900, p90: 6000 ms, p99: 6300 ms), latency samples: 208260
3. Upgrading rest of first batch to new version: e034c3de390f41cb22b77d4b09d79de0f8c5a7bd
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 5880.97 txn/s, latency: 4829.62 ms, (p50: 5400 ms, p70: 5900, p90: 6100 ms, p99: 6200 ms), latency samples: 110240
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 5967.74 txn/s, latency: 5380.42 ms, (p50: 6000 ms, p70: 6200, p90: 6400 ms, p99: 7000 ms), latency samples: 200320
4. upgrading second batch to new version: e034c3de390f41cb22b77d4b09d79de0f8c5a7bd
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 8279.19 txn/s, latency: 3394.52 ms, (p50: 3500 ms, p70: 4000, p90: 4900 ms, p99: 5300 ms), latency samples: 147060
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 8337.82 txn/s, latency: 3791.49 ms, (p50: 3300 ms, p70: 4700, p90: 5700 ms, p99: 6400 ms), latency samples: 278660
5. check swarm health
Compatibility test for 1086a5e00d773704731ab84fb4ed3538613b2250 ==> e034c3de390f41cb22b77d4b09d79de0f8c5a7bd passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking