Fixes and tests for altering topic properties #947

mmaslankaprv · 2021-03-24T13:51:13Z

Cover letter

This PR fixes handling of AlterConfigs and IncrementalAlterConfigs by allowing those requests to be called from any of the brokers. Additionally it minimizes duration between leader and follower state updates.

This PR introduces ducktape tests for altering topic configuration.

Fixes #282 #935 #689

Release notes

Full support for altering topic configurations

Release note: [1-2 sentences of what this PR changes]

src/v/cluster/service.cc

src/v/raft/state_machine.cc

src/v/kafka/server/handlers/incremental_alter_configs.cc

src/v/kafka/server/handlers/configs/config_utils.h

src/v/cluster/topics_frontend.cc

src/v/model/model.cc

emaxerrno

LGTM;

Please do minor nits and add the salient key comments on our live code-review session.

Exposed `insert_linearizable_barrier` from `raft::state_machine`. Signed-off-by: Michal Maslanka <michal@vectorized.io>

Added graceful handling of lexical cast exception thrown when parsing topic configuration property values. Signed-off-by: Michal Maslanka <michal@vectorized.io>

Fixed passing error code returned by `topics_frontend` to alter configuration responses. Signed-off-by: Michal Maslanka <michal@vectorized.io>

Added `update_topic_properties` RPC method. This is requires since both `AlterConfigs` and `IncrementalAlterConfigs` requests can be called on any broker. Since we replicate topic properties update to controller it is required to forward the update topic properties to leader controller. Signed-off-by: Michal Maslanka <michal@vectorized.io>

Implemented dispatching topic properties update to leader controller. Both `AlterConfigs` and `IncrementalAlterConfigs` can be executed on any of the nodes. We have to forward topic properties update request to current leader controller. Signed-off-by: Michal Maslanka <michal@vectorized.io>

Leveraging raft linearizable barrier to speed up applying commands on followers. In Kafka Metadata are eventually consistent. Our metadata cache and controller design leverages that fact (we do not require linearizable reads when requesting metadata). In raft followers do not know about committed index updates until then receive heartbeat following commit index update on the leader. In redpanda heartbeats are by default send every 150ms. This way follower metadata is updated with a delay of ~150ms. The delay of metadata propagation together with the fact that clients requests metadata update from arbitrary broker in the cluster may lead to situation in which clients will receive stale metadata. This is correct from the protocol perspective but it may be misleading and may cause additional delays when client backoff timeout is triggered. In order to reduce metadata propagation lag we will use `raft::insert_linearizable_barrier` method. This way we will trigger round of heartbeats immediately after data were replicated. This way we will reduce delay of metadata propagation. Signed-off-by: Michal Maslanka <michal@vectorized.io>

We have to use standard naming of topic properties. Replaced custom string with predefined constants of all topic property names. Signed-off-by: Michal Maslanka <michal@vectorized.io>

We have to make sure that all our types are presented to clients in Kafka way. Fixed printing out `model::timestamp_type`. Signed-off-by: Michal Maslanka <michal@vectorized.io>

Signed-off-by: Michal Maslanka <michal@vectorized.io>

mmaslankaprv requested review from dotnwat and emaxerrno as code owners March 24, 2021 13:51

github-actions bot added the area/redpanda label Mar 24, 2021

mmaslankaprv force-pushed the alter-config-tests branch from 43e6501 to 8b840ab Compare March 24, 2021 16:07