Status

Current state: "Accepted"

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka authorizes access to resources like topics, consumer groups etc. by way of ACLs. The current supported semantic of resource name in ACL definition is either full resource name or special wildcard '*', which matches everything.

Kafka should support a way of defining bulk ACLs instead of specifying individual ACLs.
Example use cases:

• Principal “com.company.product1.client” “user2” has access to all topics that start with “com.company.product1.”.
• Principal “com.company.client1” “user1” has access to all consumer groups that start with “com.company.client1.”.

This support will greatly simplify ACL operational story in a multi-tenant environment.

Public Interfaces

• Add new field 'ResourceNameType' to Resource and , and ResourceFilter classes to distinguish between literal and wildcard-suffix resource prefixed resource names.
  • ResourceNameType is an enum to support more types in the future.
  • ResourceNameType will include an 'ANY' type for use in filters
  • ResourceNameType will default to literal, to maintain backwards compatibility with existing clients.

• The SimpleAclAuthorizer will be enhanced to support prefixed resource names.
  
  • The contract of existing CRUD operations on the SimpleAclAuthorizer will remain the same: they will only return ACLs for the resources that exact matches the ones passed in.
  • The CRUD operations will accept literal and prefixed resources.
  • The
  Enhance implementation of getAcls(resource) in SimpleAclAuthorizer to return all matching wildcard-suffixed ACLs if 'ResourceNameType' in input resource is a wildcard-suffix.
  • The behavior of getAcls(resource) method today is to return ACLs matching only the resource literal.
  • authorize(...) method, which calls getAcls(resource) and getAcls('*') to get all the matching ACLs today.Backward compatibility will be maintained because 'ResourceNameType' defaults to literal, will instead now look for all matching literal and prefixed ACLs.

• Changes to command line tool class https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/AclCommand.scala
  • Expose a '--resource-name-type' flag which is "literal" by default to maintain backwards compatibility (though "wildcard-suffixed" is more user friendly going forward), which can be set to literal, prefixed or any, e.g.
          bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181
    2181 --add --allow-principal User:Bob --allow-principal User:Alice --allow-host
    198.51.100.0 --allow-host 198.51.100.1 --Bob  --operation Read --group my-app-* --resource-type wildcard-suffixed-name-type prefixed
  • The flag will default to 'literal', meaning the commands will return only the ACLs they previously returned, even if prefixed ACLs exists for the resource.
  • Users can now set the flag to 'all' to retrieve/delete all ACLs affecting the supplied ACLs resource.
    
    
• Changes to the admin client to support prefixed ACLs.
  • New schema version for CreateAclsRequest / DeleteAclsRequest / DescribeAclsRequest, and associated responses, which will have a new byte field
  (int)
  • in schemas to distinguish literals vs
  wildcard-suffix
  • prefixed resource names.
  Update the public documentation with the details of the new feature
  • The CreateAclRequest will only accept prefixed or literal ResourceNameTypes
  • The DeleteAclRequest and DescribeAclRequests will accept prefixed or literal, and will return delete/describe only ACLs exactly matching the resources. i.e. no pattern matching will be performed
  • The DeleteAclRequest and DescribeAclRequests will also accept a ResourceNameType of 'all', which will delete/describe all ACLs affecting the resource, including the literal wildcard ACL '*', if present.

Proposed Changes

Solution

The proposal is to extend the concept of wildcard ACL (‘*’) to support wildcard-suffixed ACLs (‘name*’)enhance the SimpleAclAuthorizer to support prefixed ACLs.
This means that it will be possible to create ACLs of type: User:clientA has READ access on topic prefixed with 'orgA* ' from 'hostA', i.e clientA has READ access to all topics that start with `orgA` from hostA.
The concept of wildcard-suffixed prefixed ACLs will be applicable only to resource names.

Storage model

Currently, ACLs are stored on ZK under path /kafka-acl/<resource-type>/<resource-name>.

For example:
ACLs for topic topicName will be stored under /kafka-acl/Topic/topicName.
ACLs for consumer group groupId will be stored under /kafka-acl/Group/groupId.

An example ACL definition looks like:

$ get /kafka-acl/Topic/topicName
{"version":1,"acls":[{"principal":"User:clientA","permissionType":"Allow","operation":"Read","host":"*"},{"principal":"User:clientA","permissionType":"Allow","operation":"Write","host":"*"},{"principal":"clientB","permissionType":"Allow","operation":"Write","host":"host1"}]}

Current supported resource names are either full resource names like topicName or a special wildcard '*'.

$ get /kafka-acl/Topic/*
{"version":1,"acls":[{"principal":"User:clientA","permissionType":"Allow","operation":"Read","host":"*"}]}
which means that clientA has read access to all topics from all hosts.

The challenge here is that some there can be both literal and prefixed resource paths with the same name. Some resources like consumer groups don't have any defined naming convention and can have '*' in their names, so can include any characters in their name. It is therefore not possible to prepend/append a 'special character' to prefixed names to distinguish them from literal ones.

We extend the same storage model to store wildcard suffix prefixed ACLs in a different location 'kafka-wildcardprefixed-acl'. Changes will first be stored at 'kafka-wildcardprefixed-acl-changes'. However, Literal resources, including the wildcard resource '*' resource name will get special treatment to support backward compatibility, will continue to be stored in their original location.

$ get /kafka-wildcardprefixed-acl/Topic/teamA*
{"version":1,"acls":[{"principal":"User:clientA","permissionType":"Allow","operation":"Read","host":"*"}]}

ACLs write path

Write to a new location 'kafka-wildcardprefixed-acl'.

$ get /kafka-wildcardprefixed-acl/Topic/orgName*
{"version":1,"acls":[{"principal":"User:clientA","permissionType":"Allow","operation":"Read","host":"*"}]}

ACLs read path

On read path, we look for all matching ACLs when :
a) getAcls(resourceWithWildcardSuffix) is called.
b) authorize(…) is called.

Access will be allowed if there is at least one ALLOW matching acl and no DENY matching acl ACL (current behavior behaviour is maintained). Note that the length of the prefix doesn't play any role here.

Matching algorithm

...

...

.

...

Compatibility, Deprecation, and Migration Plan

On downgrade, the wildcard any prefixed ACLs will be ignored because they are in separate path. This means that any wildcard prefixed ACLs will be treated as if they were never added. This is fine for ALLOW ACLs, but might have security implications if DENY ACLs are ignored.

Constructors for Resource and ResourceFilter that don't take a ResourceNameType will be deprecated in favour of those that do.

Care has been taken to ensure legacy clients can neither add, list or remove prefixed ACLs. Clients wishing to use prefixed ACLs will need to upgrade their (admin)clients.

Rejected Alternatives

• Use escaping to identify wildcard-suffix prefixed vs literals (won't work on ZK). We decided to use a separate path for wildcard-suffix prefixed ACLs