transform: multiple output topics control plane #16946
Conversation
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
This removes the last validation for preventing multiple output topics. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
rockwotj
force-pushed
the
multi-topic-control-plane
branch
from
ff87e95 to
9c8562c
Compare
| @@ -424,7 +424,11 @@ absl::flat_hash_map<ss::sstring, ss::sstring> | |||
| make_environment_vars(const model::transform_metadata& meta) { | |||
| absl::flat_hash_map<ss::sstring, ss::sstring> env = meta.environment; | |||
| env.emplace("REDPANDA_INPUT_TOPIC", meta.input_topic.tp()); | |||
| env.emplace("REDPANDA_OUTPUT_TOPIC", meta.output_topics.begin()->tp()); | |||
There was a problem hiding this comment.
Might want to throw that in the release notes for posterity
There was a problem hiding this comment.
Yeah, done. I honestly think it's probably confusing to anyone who reads that, but if they know about this random env var it will be clear. IYKYK I guess.
|
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/45823#018e1ac7-a29e-44a4-b55a-3ee7dde4e30a ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/45823#018e1ad9-9a4c-41e0-9218-b1e1f2029e07 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/45841#018e1c26-59e4-4849-9fc2-6db62763eca3 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/45841#018e1c26-59ea-4244-8fa4-4b1880043761 |
Previously the output topic was written as `REDPANDA_OUTPUT_TOPIC`, now that there can be multiple we'll write each as `REDPANDA_OUTPUT_TOPIC_%d` with the index as the key. This is technically a breaking change to remove the `REDPANDA_OUTPUT_TOPIC` key for the first output topic, but since this isn't documented anywhere and even I haven't used it it's going hurt anything here. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
rockwotj
force-pushed
the
multi-topic-control-plane
branch
from
9c8562c to
2e75f9e
Compare
|
force push: fix a broken wasi test |
The control plane already mostly supports multiple output topics, this patchset removes the last of the validation needed to deploy transforms with multiple outputs. However there are currently no SDKs with support for multiple outputs, so there isn't a way to write tests yet. Those tests will come in followups once the SDKs are updated.
Backports Required
Release Notes
Features
REDPANDA_OUTPUT_TOPICenvironment variable exposed in transforms is now removed forREDPANDA_OUTPUT_TOPIC_%dfor each output topic specified.