...

As part of the KIP-117 work to create an AdminClient for Kafka, we would like to have a way of adding, deleting, and listing the access control lists (ACLs) which are used to control access on Kafka topics and brokers.

New

...

Network Requests and Responses

Representing ACL Components on the Wire

Each ACL consists of a 4-tuple of (principal, host, operation, permission_type).

...

The "permission_type" is whether the ACL allows access or forbids it.  In the wire protocol, we represented resource_type as an INT8.  The mappings are:

• -1: none

• 0:deny

• 1: allow

Representing Resource Components on the Wire

Every ACLs is bound to a specific resource.

...

The "resource_name" is the name of the particular resource.  For example, when "resource_type" == "topic", "resource_name" will be the topic name.  In the wire protocol, we represent principal as a NULLABLE_STRING.

...

ListAclsRequest and ListAclsResponse

ListAclsRequest handles listing the ACLs in the cluster.  Principals must possess Cluster:Describe permissions to call ListAclsRequest, or be superuser.  Unauthorized requests will receive a ClusterAuthorizationException.

...

The error_code field will be set non-zero if there was an error processing the request.  If the error_code is non-zero, the resource list will be empty.

Each resource_info object describes the a specific resource, and the ACLs bound to that resource.  Note that if filters were specified in the ListAclsRequest, this may not be a complete list of all the ACLs bound to the resource, but only the ones which matched the supplied filters.

In contrast to ListAclsRequest, none of the fields in the ACL 4-tuple or the resource 2-tuple are ever set to null or none in the response.

...

AddAclsRequest and AddAclsResponse

AddAclsRequest handles adding new ACLs in the cluster. Principals must possess Cluster:All permissions to call AddAclsRequest, or be superuser.  Unauthorized requests will receive a ClusterAuthorizationException.

...

The error_code field will be non-zero if the was an error that prevented processing any part of the request.  If the error_code field is non-zero, the addition_response array will be empty.  Otherwise, there will be an addition_response for each addition in the AddAclsRequest.  These responses will be appear in the same order which the requests appeared.  If the request completed successfully, error_code will be 0 and error_string will be null.  If there was an error, the error_code will be non-zero and the error_string will give a detailed error message describing why the addition could not be performed.

...

RemoveAclsRequest and RemoveAclsResponse

RemoveAclsRequest handles removing ACLs from the cluster.  Principals must possess Cluster:All permissions to call AddAclsRequestRemoveAclsRequest, or be superuser.  Unauthorized requests will receive a ClusterAuthorizationException.

AddAclsRequestRemoveAclsRequest (Version: 0) => [resource_request]
  resource_request => resource_type resource_name [acl_request]
  resource_type => INT8
  resource_name => NULLABLE_STRING
  acl_request => principal host operation permission_type
  principal => NULLABLE_STRING
  host => NULLABLE_STRING
  operation => INT8
  permission_type => INT8 

 AddAclsRequest

RemoveAclsRequest receives a list of resource requestsfilters.  Each resource request specifies a list of requests to add particular ACLs.It will attempt to remove all the ACLs which match each filter.

Just like AddAclsRequest, RemoveAclsRequest AddAclsRequest must be sent to the controller broker.  The request is not transactional: if one addition fails, the others may proceed.  Errors Results are reported independently for each additionremoval.

AddAclsResponseRemoveAclsResponse (Version: 0) => error_code [additionfilter_response]
  error_code => INT16
  additionfilter_response => error_code [matching_acl]
  matching_acl => error_code error_string principal host operation permission_type
  error_code => INT16
  error_string => NULLABLE_STRING
  principal => STRING 
  host => STRING
  operation => INT8
  permission_type => INT8

The error_code field will be non-zero if the was an error that prevented processing any part of the request.  If the error_code field is non-zero, the addition_response array will be empty.  Otherwise, there will be an additiona removal_response for each addition filter that was passed in the AddAclsRequestRemoveAclsRequest.

Filter   These responses will be appear in the same order which the requests filters appeared.  If the request completed successfully, filter_response has a non-zero error_code will be 0 and error_string will be null.  If there was an error, the error_code will be non-zero and the error_string will give a detailed error message.

Proposed Changes

...

, that means that the filter could not be applied by the server, and the matching_acl array will be empty.  Otherwise, the matching_acl array contains a list of all the ACLs that matched the filter.  Each matching_acl will have a non-zero error code and error message if it could not be removed.

When a filters fails to match an ACLs, it is not an error.  This will simply result in getting back a filter_response with an empty matching_acl list.

New AdminClient APIs

ResourceType, AclOperation, AclPermissionType

The AdminClient needs some classes to represent the concepts of resource types, ACL operations, and permission types.

public class ResourceType {
  /**
   * Parse the given string as a resource type.
   */
  public static ResourceType fromString(String str);

  public static final ResourceType ANY = new ResourceType((byte)-1);
  public static final ResourceType TOPIC = new ResourceType((byte)0);
  public static final ResourceType GROUP = new ResourceType((byte)1);
  public static final ResourceType CLUSTER = new ResourceType((byte)2);

  private final byte ty;

  ResourceType(byte ty) {
    this.ty = ty;
  }

  public String toString();
}

public class AclOperation {
  /**
   * Parse the given string as an ACL operation.
   */
  public static AclOperation fromString(String str);

  public static final AclOperation ANY = new AclOperation((byte)-1);
  public static final AclOperation READ = new AclOperation((byte)0);
  public static final AclOperation WRITE = new AclOperation((byte)1);
  public static final AclOperation CREATE = new AclOperation((byte)2);
  public static final AclOperation DELETE = new AclOperation((byte)3);
  public static final AclOperation ALTER = new AclOperation((byte)4);
  public static final AclOperation DESCRIBE = new AclOperation((byte)5);
  public static final AclOperation CLUSTER_ACTION = new AclOperation((byte)6);
  public static final AclOperation ALL = new AclOperation((byte)7);

  private final byte op;

  AclOperation(byte op) {
    this.op = op;
  }

  public String toString();
}

public class AclPermissionType {
  /**
   * Parse the given string as a permission type.
   */
  public static AclPermissionType fromString(String str);

  public static final AclPermissionType ANY = new AclPermissionType((byte)-1);
  public static final AclPermissionType DENY = new AclPermissionType((byte)0);
  public static final AclPermissionType ALLOW = new AclPermissionType((byte)1);

  private final byte ty;

  AclPermissionType(byte ty) {
    this.ty = ty;
  }

  public String toString();
}

AdminClient#listAcls

The listAcls API surfaces ListAclsRequest.

ListAclsResult AdminClient#listAcls(AclFilter filter, ListAclsOptions options);

public ListAclsOptions 
  ListAclsOptions setTimeout(Integer timeout);
}

AclFilter objects represent the filters applied during listAcls.

Fields which are null match anything.  AclFilter.ANY matches any ACL.

public class AclFilter {
  public static final AclFilter ANY = new AclFilter(null, null, AclOperation.ANY,
                                                    AclPermissionType.ANY, ResourceType.ANY, null);
  public static final String WILDCARD = "*";
  public AclFilter(String principal, String host, AclOperation operation,
                   AclPermissionType permissionType, ResourceType resourceType, String resourceName);
  public String principal();
  public String host();
  public AclOperation Operation();
  public AclPermissionType aclPermissionType();
  public AclResourceType aclResourceType();
  public String resourceName();
}

The ListAclsResult object contains a KafkaFuture with the ACL Descriptions.

public ListAclsResult {
  public KafkaFuture<List<AclDescription>> all();
}

public class AclDescription {
  public String principal();
  public String host();
  public AclOperation Operation();
  public AclPermissionType aclPermissionType();
  public AclResourceType aclResourceType();
  public String resourceName();
  public String toString();
}

AdminClient#addAcls

The addAcls API surfaces AddAclsRequest.

AddAclsResult AdminClient#addAcls(Collection<AclDescription> acls, AddAclsOptions options);

public AddAclsOptions 
  AddAclsOptions setTimeout(Integer timeout);
}

public AddAclsResult {
  public KafkaFuture<Void> all();
  public Map<AclDescription, KafkaFuture<Void>> results();
}

AdminClient#removeAcls

The removeAcls API surfaces RemoveAclsRequest.

RemoveAclsResult AdminClient#removeAcls(Collection<AclFilter> filters, RemoveAclsOptions options);

public RemoveAclsOptions 
  RemoveAclsOptions setTimeout(Integer timeout);
}

public RemoveAclsResult {
  public KafkaFuture<Map<AclFilter, List<AclDescription>> all();
  public Map<AclFilter, List<KafkaFuture<AclDescription>>> results();
}

Note that removing a topic does not remove the associated ACLs, nor does removing ACLs remove the associated topic.

Migration Plan

Once AdminClient supports ACL operations, we can transition the command-line utilities to using it, instead of contacting ZooKeeper directly.

Compatibility Plan

Since there are no existing ACL APIs and requests, backwards compatibility is not an issue.  However, we still need to think about forwards compatibility.  The version of the AdminClient that we release in 0.11 should be able to interact with future versions of the broker.

If we later add new resource types, operation types, and so forth, we would like to be able to interact with them with the old AdminClient.  This is why the AclResourceType, AclOperation, and AclPermissionType classes are not enums.  If we get an INT8 which we don't know the name for, we can still create a Java object with that byte.  That Java object will describe itself as UNKNOWN(9) instead of WRITE.

What if we later add more dimensions to the 4-tuple that describes ACLs, or the 2-tuple that describes resources?  AddAcls will continue to work, although the entries it creates will always get the default value for the new dimension.  ListAcls and DeleteAcls will also continue to work.  The filters created by older clients will always have an implicit "any" entry for the new dimension.  This allows the old AdminClient to continue to be able to function in the new environment.

Rejected Alternatives

Combined AddAcls and RemoveAcls API (AlterAcls)

We could have combined AddAcls and RemoveAcls into a single AlterAcls RPC.  However, this would have been a bad idea for a few reasons:

• The name "AlterAcls" suggests that ACLs are being altered.  However, in fact ACLs are only being added or removed, but not altered.
• It's unclear what order the add and remove operations happen in.
• It is unclear whether a remove operation can remove something added in the same AlterAcls request.
• If add and remove operations are reordered, a security hole could be created when brokers are configured with default-allow behavior.  Deleting a restrictive ACL on a secure topic before adding a new restrictive ACL on that topic creates a window of vulnerability.
• AddAcls and RemoveAcls is similar to the existing AddTopics and RemoveTopics APIs.

  public AclFilter ANY = new AclFilter();