Test datalake with recovery mode and disabled partitions

[ztlpn]

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v24.3.x
• v24.2.x
• v24.1.x

Release Notes

Improvements

• Disable datalake services in recovery mode

[dotnwat]

/ci-repeat 5
skip-units
dt-repeat=100
tests/rptest/tests/datalake/recovery_mode_test.py

[vbotbuildovich]

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/59319#01939901-3ee1-4f93-b9c7-9a656977bebb:

"rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_disabled_partitions.cloud_storage_type=CloudStorageType.S3.filesystem_catalog_mode=False"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/59319#01939901-3ee0-4d2d-a5fe-3b0edf16105b:

"rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_disabled_partitions.cloud_storage_type=CloudStorageType.S3.filesystem_catalog_mode=False"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/59319#01939905-17d7-4520-a24f-0ba7df24d931:

"rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_disabled_partitions.cloud_storage_type=CloudStorageType.S3.filesystem_catalog_mode=True"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/59319#01939901-3ee2-4b3d-b9a7-4e9a669e9249:

"rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_disabled_partitions.cloud_storage_type=CloudStorageType.S3.filesystem_catalog_mode=False"

[vbotbuildovich]

Retry command for Build#59319

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/datalake/recovery_mode_test.py::DatalakeRecoveryModeTest.test_disabled_partitions@{"cloud_storage_type":1,"filesystem_catalog_mode":false}
tests/rptest/tests/datalake/recovery_mode_test.py::DatalakeRecoveryModeTest.test_disabled_partitions@{"cloud_storage_type":1,"filesystem_catalog_mode":true}

[ztlpn]

Some

rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-2(1) example="ERROR 2024-12-05 23:16:14,283 [shard 1:data] s3 - util.cc:108 - Unexpected error seastar::nested_exception: std::__1::system_error (error system:32, sendmsg: Broken pipe) (while cleaning up after std::__1::system_error (error system:32, sendmsg: Broken pipe))">

Otherwise no errors.

[dotnwat]

looks like a merge conflict in application.cc

[andrwng]

nit: maybe produce to the topic to make it more likely that there's more data that would be scheduled if not for recovery on the given node? And maybe again to "foo" and "bar" after enabling recovery mode for the cluster? At that point, maybe we could also check that the table doesn't grow before and after the sleep

[andrwng]

Same in the next test -- or does recovery mode prevent us from producing?

[ztlpn]

does recovery mode prevent us from producing?

Yes exactly. I thought about adding some checks that the table doesn't grow in recovery mode, but they would be either trivial or inherently flaky.

[vbotbuildovich]

/backport v24.3.x

[vbotbuildovich]

Failed to create a backport PR to v24.3.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-24446-v24.3.x-539 remotes/upstream/v24.3.x
git cherry-pick -x 7cfbcb05d2 159a2722d7 80d3ac2076

Workflow run logs.