Status

Current state: Under discussion

Discussion thread:

JIRA: Unable to render Jira issues macro, execution error.

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

A number of Admin APIs have been/will be introduced as part of KIP-4. A subset of these Admin APIs allow the user to create, update or delete cluster resources: create topics, delete topics, alter topics, alter ACLs and alter configs. Create and delete topics were added to Kafka in 0.10.0.0 while the others are still in progress. These APIs are a major step towards self-serve Kafka clusters where application developers can, for example, create topics without having to go through the admins/operations team and without access to ZooKeeper.

There is, however, no way to validate the user request in order to restrict the operation parameters. The operations team may want to enforce that the replication factor, min.insync.replicas and/or retention settings for a topic are within an allowable range for example. In this KIP, we limit ourselves to validation of the create topic request, but the approach can be used for other request types in the future.

Public Interfaces

A user can define a policy manager similar to the pluggable Authorizer by setting create.topics.policy.class.name in server.properties and implementing the the CreateTopicPolicy interface. The interface will live in the clients jar under the org.apache.kafka.server.policy package.

package org.apache.kafka.server.policy;


public interface CreateTopicPolicy {
  void validate(TopicDetails topicDetails) throws InvalidRequestException;
}

public class TopicDetails {
    private final String topic;
    private final int numPartitions;
    private final short replicationFactor;
    private final Map<Integer, List<Integer>> replicasAssignments;
    private final Map<String, String> configs;


    public TopicDetails(String topic, int numPartitions, short replicationFactor, Map<Integer, List<Integer>> replicasAssignments, Map<String, String> configs) {
        this.topic = topic;
        this.numPartitions = numPartitions;
        this.replicationFactor = replicationFactor;
        this.replicasAssignments = replicasAssignments;
        this.configs = configs;
	}
}

Users will have to ensure that their code is in the broker's classpath.

Proposed Changes

AdminManager will instantiate a CreateTopicsPolicy instance if create.topics.policy.class.name is defined and will call the validate method after existing checks are done (i.e. numPartitions and replicationFactor cannot be used with replicaAssignments). If a request with does not pass validation, an InvalidRequestException is thrown (in the same way as existing checks). One limitation is that there is no mechanism to pass the error message back to the client. In order to do this, we would have to modify the response to include an error string.

As described in the previous section, we are proposing one policy config per supported request type. The main advantage is that we can add additional configs in a compatible manner. Implementors can use a single implementation class if they wish, but they would have to set all the relevant configs. Also, there would be one interface per supported request type. Once we move to Java 8, we can provide an interface that inherits from the various interfaces with an empty implementation via default methods.

Compatibility, Deprecation, and Migration Plan

There is no compatibility impact as there is no change in behaviour unless the new config is used.

Rejected Alternatives

1. A single config for an implementation of an interface with multiple validate methods: there would be no way to add methods without breaking binary compatibility until we move to Java 8.
2. Restrict it via configs instead of pluggable interfaces: it would be clunky to provide configs for every requirement that users may have.
3. Extending Authorizer to perform validation: even though this could work, it's not particularly intuitive.