roachtest: disk-stalled/wal-failover/among-stores failed

[cockroach-teamcity]

roachtest.disk-stalled/wal-failover/among-stores failed with artifacts on release-24.1 @ 3111a51bdb2cbf889c92f4440bf88b0eb682cb96:

(disk_stall.go:174).runDiskStalledWALFailover: unexpectedly high p99.99 latency 1.154394337s at 2024-04-21T13:17:00Z
(cluster.go:2348).Run: context canceled
test artifacts and logs in: /artifacts/disk-stalled/wal-failover/among-stores/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=true
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=2

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: disk-stalled/wal-failover/among-stores failed #122364 roachtest: disk-stalled/wal-failover/among-stores failed [C-test-failure O-roachtest O-robot T-storage branch-master]

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

Jira issue: CRDB-38060

[cockroach-teamcity]

roachtest.disk-stalled/wal-failover/among-stores failed with artifacts on release-24.1 @ 2873eb728d555695bf5a392d748142e9b05cd36d:

(disk_stall.go:174).runDiskStalledWALFailover: unexpectedly high p99.99 latency 1.021233281s at 2024-04-23T13:23:00Z
(cluster.go:2348).Run: context canceled
test artifacts and logs in: /artifacts/disk-stalled/wal-failover/among-stores/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=true
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=2

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: disk-stalled/wal-failover/among-stores failed #122364 roachtest: disk-stalled/wal-failover/among-stores failed [C-test-failure O-roachtest O-robot T-storage branch-master]

This test on roachdash | Improve this report!

[sumeerbhola]

Oddly, there are no node logs from the last failure here, or in #122364.

13:58:08 test_impl.go:414: test failure #1: full stack retained in failure_1.log: (disk_stall.go:174).runDiskStalledWALFailover: unexpectedly high p99.99 latency 1.021233281s at 2024-04-23T13:23:00Z

SQL and KV latency were high, but raft log commit latency did not spike at p100. Suggests something observed the disk stall in the KV code.

KV qps was unaffected, which is consistent with flushed bytes being unaffected, but compactions fell, which is odd.

[sumeerbhola]

closing this as a duplicate of #122364