-
Notifications
You must be signed in to change notification settings - Fork 651
Project Admin APIs
The current API exposes some metadata about the cluster, such as the list of brokers and the list of topics with partition replica assignment information. However, this it is currently read-only. This project will add a set of administrative APIs that will allow making modifications to the cluster configuration, as well as fill out some missing requests or data in responses that didn't make it into the first version.
The motivation for adding admin APIs is to make managing a Kafka cluster significantly more accessible. Currently the only interface for administrative tasks are the command line tools included with Kafka. While straightforward, these are somewhat painful to use (just count the number of times you need to specify zookeeper or broker information). A REST API won't fix this directly -- replacing command line tools with cURL requests isn't an improvement -- but it makes it much simpler to build better tools using the REST API without those tools having to reach into Kafka internals.
This is still an open issue (with work in progress) in Kafka itself. The problem is perhaps compounded by the addition of the REST proxy since it may need a separate authentication and authorization mechanism. This includes a) defining who agents are, b) specifying how they are authenticated, c) specifying how authorization policies are defined, stored, and applied.
The design here is still uncertain, but here are a few possibilities:
- Do nothing for now and wait until Kafka's solution is clearer to ensure we can work with it.
- Try to track the ongoing TLS and SASL work, target that specifically, and use proxy authorization to interact with the Kafka cluster.
- Alternatively, select a single, standard, baked-in solution that is different from Kafka's but fits the environment better. We'd need to figure out how to bridge it to Kafka.
- Make it all pluggable, and provide at least one usable implementation. At it's core, this option just means we need to provide a few hooks to where the plugin can inspect the HTTP request and tell us whether the caller has permission for the action and any changes that need to be made when sending commands to brokers to apply the admin action (e.g. using TLS/SASL/Kerberos).
- Punt entirely and expect that operators will run the proxy itself behind another proxy (though probably on the same host) that handles all the auth*n. If necessary, we could place all the admin functionality under a separate /admin namespace so it would be easy to apply a simple URL-based policy.
Until we do all admin operations via the brokers, however, it's unclear how much value this adds. On the one hand, we'll still need Zookeeper access. On the other, the REST proxy could offer a migration path to locking things down: first convert to using REST proxy for all admin actions, then lock down access to ZK to only REST proxy nodes.
There are a number of outstanding KIPs which might affect both the design and implementation of this project:
- KIP-4 - Command line and centralized administrative operations
- Adds controller metadata.
- New admin wire protocol for pretty much all the interesting admin actions
- KIP-5 - Broker Configuration Management
- Enables dynamic broker config by adding new protocol, stores them in ZK
- KIP-6 - New reassignment partition logic for rebalancing
- Adds new algorithm and CLI options for reassignment
- Could affect how much functionality we expose (e.g. might include a simpler, fully automated version based on this tool)
- Not critical to make use of it immediately, but should make sure we leave room for in the API for it
- KIP-7 - Security - IP Filtering
- Possibly relevant to authentication/authorization.
- KIP-13 - Quota Design
- We'll need to include APIs for this - it is new API you wouldn't find by looking at the existing CLI tools.
- May need new new resource for this since it isn't obviously attached to any existing resources.
- May need to be able to pick up current quota % via JMX in order to provide feedback about quota limits via this API.
KIP-4 specifically needs to be highlighted. It is likely to drastically affect the implementation of this project because it will add admin actions to the brokers themselves, which will eventually allow us to use the standard Kafka protocol to implement all this functionality. Today, we have to use Kafka internals (or reimplementations/modifications of them) and work directly with ZK to implement a lot of this functionality.
However, KIP-4 is not yet approved and likely depends on some other changes before it can be implemented. This may block implementation of this project. One option would be to implement this immediately and later migrate over to the KIP-4 protocols instead of mucking with Kafka internals. Doing so likely means any Kafka-based auth*n cannot be implemented until KIP-4 work is done.
Regardless of whether KIP-4 blocks work on this project, we need to consider its changes in the API design. Most significantly, it introduces async admin commands: all commands are going to issue responses immediately, but may not be complete yet. We could either choose to keep the REST proxy versions simple and synchronous, or expose the nearly-raw versions of the commands such that the user needs to issue subsequent requests to discover the final state, or we could provide both.
Note Currently this is an overview of the API additions we'll need. Each one needs to be expanded to include proposed request and response formats. Ideally this section only needs minor changes before being included in the API documentation.
GET /brokers/- We should have something here, but configs aren't available. This may just include the partition assignments for the broker for now.
-
PUT /brokers/<broker_id>? - Configs aren't dynamic currently, so unless KIP-5 makes it through, this may not make sense for this first iteration.
- May need to break this down into
/brokers/<broker_id>/assignmentsand/brokers/<broker_id>/configs.
GET /topics/<topic>- Already exists, but may need to add fields to response
POST/PUT /topics/<topic>/config- Updates configs for this topic
POST /topics/<topic>/partitions- Increase number of partitions for this topic
GET /topics/<topic>/partitions/<partition>- Already exists, but may need to add fields to response
GET /consumers- Equivalent to
ConsumerGroupCommand --list GET /consumers/<consumer>- Equivalent to
ConsumerGroupCommand --describe PUT /consumers/<consumer>/offsets- Allows manually writing offsets. This may need
/topics/<topic>/partitions/<partition>in its path, although how this is handled also depends on how we expose wildcard consumers. DELETE /consumers/<consumer>
This would be a new top-level resource. The motivation is to handle changes that don't target just one of the existing resources.
GET /cluster/controller- Returns the current controller. Could either return the ID, or might include additional data from the controller. See KIP-4 metadata addition.
GET/POST /cluster/assignments- Partition reassignment isn't necessarily topic or partition specific, so it needs to be under this cluster resource.
- We need to think about how to map all the actions that the repartitioning tool will be able to do after KIP-6 (and beyond).
- This can be long running, we'll need some sort of async interface.
-
POST ?- Preferred replica leader election - This is an action, so it's not clear what resource is should be on. maybe
POST /cluster/leaders? -
GET/PUT /cluster/quotas/<app_id>- The exact URL format will depend on the final form of KIP-13.
One final note: there are Import/Export ZK Offset tools (broker partition offsets). I'm not sure when people actually use these and whether the aggregate form (rather than getting them from each broker's resource) is useful.
Most projects need to address what compatibility issues might arise due to their changes. However, since this project involves only additions to the API, there are no compatibility concerns.