[CORE-7803] Audit Log Manager: Refactoring - use `client::produce_record_batch` and reduce retries. #23775

oleiman · 2024-10-14T16:06:38Z

TODO:

better error handling
bazel stuff
make a run at broker_error repro

Backports Required

Release Notes

Bug Fixes

Fixes a bug where audit log manager would retry a bad request forever, causing buffers to fill up, blocking audit log appends and preventing authZ.

oleiman · 2024-10-14T16:09:01Z

scale test result (passed):
audit_stress_result.2024.10.11.tar.gz

It's not immediately clear to me a) how the stress test error bounds were derived or b) whether we might have regressed performance at all.

vbotbuildovich · 2024-10-15T01:38:50Z

vbotbuildovich · 2024-10-15T01:39:13Z

Retry command for Build#56495

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/cloud_storage_timing_stress_test.py::CloudStorageTimingStressTest.test_cloud_storage@{"cleanup_policy":"delete"}

michael-redpanda

looks really good! At least one must change (boolean logic check)

src/v/kafka/client/BUILD

src/v/kafka/client/record_batcher.cc

src/v/security/audit/CMakeLists.txt

src/v/security/audit/audit_log_manager.cc

oleiman · 2024-10-15T21:30:34Z

force push contents:

optionally inject a ss::logger into record_batcher
various CR feedback on audit changes

vbotbuildovich · 2024-10-16T00:54:42Z

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/56560#01929272-4461-4115-a479-ce0760154bc3:

"rptest.tests.topic_recovery_test.TopicRecoveryTest.test_fast3.cloud_storage_type=CloudStorageType.ABS"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/56770#01929c71-7831-4cb2-ae4b-04f0913a7970:

"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.executed.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.prepared.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.out.stage=.executed.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=None.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.executing.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.preparing.use_alias=False"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/56770#01929c84-b709-443f-b75d-02f64b77d224:

"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.executed.use_alias=True"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.prepared.use_alias=True"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.out.stage=.executing.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=CancellationStage.dir=.out.stage=.preparing.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=None.use_alias=True"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.executing.use_alias=True"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.preparing.use_alias=True"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/56770#01929c84-b711-4676-b0a3-9591c7568fa8:

"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.prepared.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.executed.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.out.stage=.executed.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=None.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.executing.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.preparing.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=CancellationStage.dir=.out.stage=.prepared.use_alias=False"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/56770#01929c84-b70c-455a-a8e1-a1789f798894:

"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.executing.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.preparing.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.out.stage=.prepared.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.executed.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.prepared.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=CancellationStage.dir=.out.stage=.executed.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=None.use_alias=False"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/56770#01929c84-b70f-4105-8f55-7b8694146d6a:

"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.executing.use_alias=True"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.out.stage=.preparing.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.preparing.use_alias=True"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.executed.use_alias=True"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.prepared.use_alias=True"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=CancellationStage.dir=.out.stage=.executing.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=None.use_alias=True"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/56770#01929c71-7832-4173-b9e2-c9e947c53f77:

"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.executed.use_alias=True"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.prepared.use_alias=True"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.out.stage=.executing.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=None.use_alias=True"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.preparing.use_alias=True"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.executing.use_alias=True"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/56770#01929c71-7833-4f97-8979-482cc2f693ff:

"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.executing.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.preparing.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.out.stage=.prepared.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.executed.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.prepared.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=CancellationStage.dir=.out.stage=.executed.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=None.use_alias=False"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/56770#01929c71-7830-41ca-bb3c-e8d5271b2f03:

"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.executing.use_alias=True"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.preparing.use_alias=True"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=TmtpdiParams.cancellation=CancellationStage.dir=.out.stage=.preparing.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.executed.use_alias=True"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=CancellationStage.dir=.in.stage=.prepared.use_alias=True"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=CancellationStage.dir=.out.stage=.executing.use_alias=False"
"rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=TmtpdiParams.cancellation=None.use_alias=True"

vbotbuildovich · 2024-10-16T00:55:00Z

Retry command for Build#56560

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_fast3@{"cloud_storage_type":2}
tests/rptest/transactions/transactions_test.py::TxUpgradeRevertTest.test_snapshot_compatibility

oleiman · 2024-10-16T15:23:28Z

/ci-repeat 1
release
skip-redpanda-build
skip-units
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_fast3@{"cloud_storage_type":2}
tests/rptest/transactions/transactions_test.py::TxUpgradeRevertTest.test_snapshot_compatibility

oleiman · 2024-10-16T17:25:55Z

CI Failures:

bazel build - gazelle
https://redpandadata.atlassian.net/browse/CORE-7943

michael-redpanda

lgtm, ❤️

oleiman · 2024-10-16T19:17:28Z

/cdt
dt-repeat=3
tests/rptest/scale_tests/audit_log_test.py

oleiman · 2024-10-16T20:53:47Z

CDT failure looks like infra https://buildkite.com/redpanda/redpanda/builds/56658#019296c5-6c31-4c3d-9a2b-caec09836335

oleiman · 2024-10-16T20:54:08Z

/cdt
tests/rptest/scale_tests/audit_log_test.py

oleiman · 2024-10-16T22:01:29Z

/ci-repeat 1
skip-redpanda-build
skip-units
tests/rptest/transactions/transactions_test.py::TxUpgradeRevertTest.test_snapshot_compatibility

oleiman · 2024-10-16T22:25:52Z

CI Failures:

build-debug isn't real - ran a specific test that doesn't get run in debug, so fails on empty/no report. had a passing debug build previously
build-release known issue https://redpandadata.atlassian.net/browse/CORE-7943

vbotbuildovich · 2024-10-16T22:42:45Z

Retry command for Build#56675

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/transactions/transactions_test.py::TxUpgradeRevertTest.test_snapshot_compatibility

src/v/kafka/client/CMakeLists.txt

src/v/kafka/client/test/record_batcher_test.cc

src/v/kafka/client/test/utils.cc

BenPope · 2024-10-17T12:42:00Z

src/v/kafka/client/record_batcher.cc

+      size_t batch_max_bytes, std::optional<ss::logger*> log)
+      : _batch_max_bytes(batch_max_bytes)
+      , _log(log.value_or(&kclog)) {
+        vassert(_log != nullptr, "Injected logger must not be nullptr");


Thought: Do we have a null logger? I.e., explicit nullptr means don't log?

No clue, but awesome idea

src/v/security/audit/audit_log_manager.h

src/v/security/audit/audit_log_manager.cc

Useful in audit_log_manager as well as transform logging Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

And adds make_batch_of_one Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

We need these for sorting out which partitions are locally led Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Previous implementation used a very high value for retries on the internal kafka client, which prevents the client from recovering certain types of errors. Instead, we batch up drained records on the manager side, allowing us to hold a copy of each batch in memory and retry failed produce calls from "scratch". This also allows us to be _much_ more aggressive about batching. The internal kafka client will calculate a destination partition for each record, round robin style over the number of partitions. In the new scheme, we shoot for a maximally sized batch first, then select a destination, still round-robin style, but biasing heavily toward locally led partitions. In this way, given the default audit per-shard queue limit and default max batch size (both 1MiB), the most common drain operation should result in exactly one produce request. Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

oleiman · 2024-10-17T20:31:37Z

force push contents:

CR feedback
fix merge conflict & add signoffs

vbotbuildovich · 2024-10-17T23:30:07Z

Retry command for Build#56770

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["in","executed"],true],"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["in","prepared"],true],"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["out","executing"],false],"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["out","preparing"],false],"transfer_leadership":true}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[null,true],"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["in","executing"],true],"transfer_leadership":true}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["in","preparing"],true],"transfer_leadership":true}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["in","executed"],false],"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["in","prepared"],false],"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["out","executed"],false],"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[null,false],"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["in","executing"],false],"transfer_leadership":true}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["in","preparing"],false],"transfer_leadership":true}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["out","prepared"],false],"transfer_leadership":true}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["in","executing"],false],"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["in","preparing"],false],"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["out","prepared"],false],"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["in","executed"],false],"transfer_leadership":true}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["in","prepared"],false],"transfer_leadership":true}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["out","executed"],false],"transfer_leadership":true}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[null,false],"transfer_leadership":true}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["in","executing"],true],"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["out","preparing"],false],"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["in","preparing"],true],"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["in","executed"],true],"transfer_leadership":true}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["in","prepared"],true],"transfer_leadership":true}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[["out","executing"],false],"transfer_leadership":true}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":[null,true],"transfer_leadership":true}

Followup to redpanda-data#23775

oleiman · 2024-10-18T04:18:51Z

CI Failures:

build-debug ; build-release same issue, seems maybe related to https://redpandadata.atlassian.net/browse/CORE-7930 but not exactly. not related to this PR

oleiman · 2024-10-20T03:53:54Z

/ci-repeat 1
skip-redpanda-build
skip-units
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity

oleiman · 2024-10-20T22:39:39Z

CI Failure:

bazel-build tcp connection timed out at link time? presumably a remote object cache issue?

vbotbuildovich · 2024-10-21T15:56:52Z

/backport v24.2.x

vbotbuildovich · 2024-10-21T15:56:53Z

/backport v24.1.x

vbotbuildovich · 2024-10-21T15:58:00Z

Failed to create a backport PR to v24.1.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-23775-v24.1.x-390 remotes/upstream/v24.1.x
git cherry-pick -x 090f41b02f 11b8a23255 37414a7d58 5c7ed51bff e5fd326728

Workflow run logs.

vbotbuildovich · 2024-10-21T15:58:05Z

Failed to create a backport PR to v24.2.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-23775-v24.2.x-111 remotes/upstream/v24.2.x
git cherry-pick -x 090f41b02f 11b8a23255 37414a7d58 5c7ed51bff e5fd326728

Workflow run logs.

Followup to redpanda-data#23775

oleiman self-assigned this Oct 14, 2024

github-actions bot added area/build area/redpanda area/wasm WASM Data Transforms labels Oct 14, 2024

oleiman changed the title ~~Audit Log Manager: Refactoring - use client::produce_record_batch and reduce retries.~~ [CORE-7803] Audit Log Manager: Refactoring - use client::produce_record_batch and reduce retries. Oct 14, 2024

oleiman mentioned this pull request Oct 14, 2024

[CORE-7803] Refactor audit_log_manager to perform batching and partition assignment outside kafka::client #23757

Closed

10 tasks

oleiman force-pushed the audit/core-7803/audit-client-reduce-retries branch from 22dfb96 to bb95d56 Compare October 14, 2024 22:44

oleiman marked this pull request as ready for review October 14, 2024 22:44

oleiman requested review from a team, BenPope and michael-redpanda and removed request for a team October 14, 2024 23:05

michael-redpanda reviewed Oct 15, 2024

View reviewed changes

oleiman force-pushed the audit/core-7803/audit-client-reduce-retries branch from bb95d56 to 0d67052 Compare October 15, 2024 21:29

oleiman requested a review from michael-redpanda October 16, 2024 15:23

michael-redpanda previously approved these changes Oct 16, 2024

View reviewed changes

BenPope reviewed Oct 17, 2024

View reviewed changes

oleiman dismissed michael-redpanda’s stale review via 6d39098 October 17, 2024 19:29

oleiman force-pushed the audit/core-7803/audit-client-reduce-retries branch from 0d67052 to 6d39098 Compare October 17, 2024 19:29

oleiman added 5 commits October 17, 2024 13:28

record_batcher: Move from transform/logging to kafka/client

090f41b

Useful in audit_log_manager as well as transform logging Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

k/record_batcher: Make k/v interfaces optional<iobuf>

11b8a23

And adds make_batch_of_one Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

k/record_batcher: Optionally inject a ss::logger

37414a7

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

audit: Plumb metadata_cache and node_id into audit_log_manager

5c7ed51

We need these for sorting out which partitions are locally led Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

oleiman force-pushed the audit/core-7803/audit-client-reduce-retries branch from 6d39098 to e5fd326 Compare October 17, 2024 20:29

oleiman added a commit to oleiman/redpanda that referenced this pull request Oct 17, 2024

k/record_batcher: Move to kafka/data

3417b42

Followup to redpanda-data#23775

oleiman mentioned this pull request Oct 17, 2024

k/record_batcher: Move to kafka/data #23832

Merged

7 tasks

oleiman requested review from BenPope and michael-redpanda October 18, 2024 17:25

BenPope approved these changes Oct 21, 2024

View reviewed changes

oleiman merged commit 045fa65 into redpanda-data:dev Oct 21, 2024
17 checks passed

vbotbuildovich mentioned this pull request Oct 21, 2024

[v24.1.x] [CORE-7803] Audit Log Manager: Refactoring - use client::produce_record_batch and reduce retries. #23863

Closed

vbotbuildovich mentioned this pull request Oct 21, 2024

[v24.2.x] [CORE-7803] Audit Log Manager: Refactoring - use client::produce_record_batch and reduce retries. #23864

Closed

oleiman added a commit to oleiman/redpanda that referenced this pull request Oct 21, 2024

k/record_batcher: Move to kafka/data

0c70484

Followup to redpanda-data#23775

This was referenced Oct 21, 2024

[v24.2.x] [CORE-7803] Audit Log Manager: Refactoring - use and reduce retries. #23867

Merged

[v24.1.x] [CORE-7803] Audit Log Manager: Refactoring - use and reduce retries. #23868

Merged

oleiman added a commit to oleiman/redpanda that referenced this pull request Oct 28, 2024

k/record_batcher: Move to kafka/data

3ddb1e3

Followup to redpanda-data#23775

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CORE-7803] Audit Log Manager: Refactoring - use `client::produce_record_batch` and reduce retries. #23775

[CORE-7803] Audit Log Manager: Refactoring - use `client::produce_record_batch` and reduce retries. #23775

oleiman commented Oct 14, 2024 •

edited

Loading

oleiman commented Oct 14, 2024

vbotbuildovich commented Oct 15, 2024 •

edited

Loading

vbotbuildovich commented Oct 15, 2024

michael-redpanda left a comment

oleiman commented Oct 15, 2024

vbotbuildovich commented Oct 16, 2024 •

edited

Loading

vbotbuildovich commented Oct 16, 2024 •

edited

Loading

oleiman commented Oct 16, 2024

oleiman commented Oct 16, 2024

michael-redpanda left a comment

oleiman commented Oct 16, 2024

oleiman commented Oct 16, 2024

oleiman commented Oct 16, 2024

oleiman commented Oct 16, 2024

oleiman commented Oct 16, 2024 •

edited

Loading

vbotbuildovich commented Oct 16, 2024

BenPope Oct 17, 2024

oleiman Oct 17, 2024

oleiman commented Oct 17, 2024

vbotbuildovich commented Oct 17, 2024 •

edited

Loading

oleiman commented Oct 18, 2024

oleiman commented Oct 20, 2024

oleiman commented Oct 20, 2024 •

edited

Loading

vbotbuildovich commented Oct 21, 2024

vbotbuildovich commented Oct 21, 2024

vbotbuildovich commented Oct 21, 2024

vbotbuildovich commented Oct 21, 2024

[CORE-7803] Audit Log Manager: Refactoring - use client::produce_record_batch and reduce retries. #23775

[CORE-7803] Audit Log Manager: Refactoring - use client::produce_record_batch and reduce retries. #23775

Conversation

oleiman commented Oct 14, 2024 • edited Loading

Backports Required

Release Notes

Bug Fixes

oleiman commented Oct 14, 2024

vbotbuildovich commented Oct 15, 2024 • edited Loading

vbotbuildovich commented Oct 15, 2024

Retry command for Build#56495

michael-redpanda left a comment

Choose a reason for hiding this comment

oleiman commented Oct 15, 2024

vbotbuildovich commented Oct 16, 2024 • edited Loading

vbotbuildovich commented Oct 16, 2024 • edited Loading

Retry command for Build#56560

oleiman commented Oct 16, 2024

oleiman commented Oct 16, 2024

michael-redpanda left a comment

Choose a reason for hiding this comment

oleiman commented Oct 16, 2024

oleiman commented Oct 16, 2024

oleiman commented Oct 16, 2024

oleiman commented Oct 16, 2024

oleiman commented Oct 16, 2024 • edited Loading

vbotbuildovich commented Oct 16, 2024

Retry command for Build#56675

BenPope Oct 17, 2024

Choose a reason for hiding this comment

oleiman Oct 17, 2024

Choose a reason for hiding this comment

oleiman commented Oct 17, 2024

vbotbuildovich commented Oct 17, 2024 • edited Loading

Retry command for Build#56770

oleiman commented Oct 18, 2024

oleiman commented Oct 20, 2024

oleiman commented Oct 20, 2024 • edited Loading

vbotbuildovich commented Oct 21, 2024

vbotbuildovich commented Oct 21, 2024

vbotbuildovich commented Oct 21, 2024

vbotbuildovich commented Oct 21, 2024

[CORE-7803] Audit Log Manager: Refactoring - use `client::produce_record_batch` and reduce retries. #23775

[CORE-7803] Audit Log Manager: Refactoring - use `client::produce_record_batch` and reduce retries. #23775

oleiman commented Oct 14, 2024 •

edited

Loading

vbotbuildovich commented Oct 15, 2024 •

edited

Loading

vbotbuildovich commented Oct 16, 2024 •

edited

Loading

vbotbuildovich commented Oct 16, 2024 •

edited

Loading

oleiman commented Oct 16, 2024 •

edited

Loading

vbotbuildovich commented Oct 17, 2024 •

edited

Loading

oleiman commented Oct 20, 2024 •

edited

Loading