Status

Current state: Under Discussion

Discussion thread: 

JIRA: 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Currently, when using KRaft mode, users still have to have an Apache ZooKeeper instance if they want to use AclAuthorizer. We should have a built-in Authorizer for KRaft mode that does not depend on ZooKeeper.

Public Interfaces

Classes

org.apache.kafka.metadata.authorizer.StandardAuthorizer

The StandardAuthorizer is a new Authorizer class which stores its ACLs in the __cluster_metadata topic. It is used by default in KRaft clusters if the authorizer.class.name configuration is not set. Note that the default for ZooKeeper-based clusters remains the existing AclAuthorizer class.

In general, StandardAuthorizer should act as a drop-in replacement for AclAuthorizer, implementing all the same behaviors. It will also support the same static configuration keys as AclAuthorizer, including super.users (a set of users that have all permissions) and allow.everyone.if.no.acl.found.

Records

AccessControlEntryRecord

This record stores information about access controls that have been put in place in the cluster.

This is basically the same information that is stored in AclBinding, except that it contains a unique ID (UUID) to identify the ACL.

When an AccessControlEntryRecord appears in the log, it signals that a new ACL has been created.

When an AccessControlEntryRecord appears in a snapshot, it describes an ACL that exists in the cluster. The ACLs in the snapshot do not appear in any deterministic order.

{
  "apiKey": ...,
  "type": "metadata",
  "name": "AccessControlEntryRecord",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "Id", "type": "uuid", "versions": "0+",
      "about": "The unique ID of this ACL." },
    { "name": "ResourceType", "type": "int8", "versions": "0+",
      "about": "The resource type." },
    { "name": "ResourceName", "type": "string", "versions": "0+",
      "about": "The resource name." },
    { "name": "PatternType", "type": "int8", "versions": "0+",
      "about": "The resource name pattern type." },
    { "name": "Principal", "type": "string", "versions": "0+",
      "about": "The principal name." },
    { "name": "Host", "type": "string", "versions": "0+",
      "about": "The host name." },
    { "name": "Operation", "type": "int8", "versions": "0+",
      "about": "The AclOperation." },
    { "name": "PermissionType", "type": "int8", "versions": "0+",
      "about": "The AclPermissionType." }
  ]
}

RemoveAccessControlRecord

When it appears in the metadata log, this record indicates that an ACL has been deleted. This record does not appear in the metadata snapshot.

{
  "apiKey": ...,
  "type": "metadata",
  "name": "RemoveAccessControlEntryRecord",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "Id", "type": "uuid", "versions": "0+",
      "about": "The ID of the AccessControlEntry to remove." }
  ]
}

Metadata Shell

The metadata shell will support examining KRaft ACLs. Each ACL will appear in /acl/id/<uuid> in its JSON form.

Design

Contexts

StandardAuthorizer runs in two contexts: on the broker, and in the controller. Just like currently, a node running in combined mode (broker+controller) will have two separate Authorizer instances: one for the broker, and one for the controller.

In most cases, these Authorizer objects should have the same state. However, when the controller is active, its Authorizer state may be slightly ahead of the the broker's Authorizer state. This will happen in the time between when a new ACL is created (or deleted) and the time that this change is persisted durably by a majority of controller quorum peers.

Atomic Snapshot Load

Unlike many other Kafka components, the Authorizer is accessed from many different threads without locking. Any change to the Authorizer state will be reflected immediately in those threads. Therefore, when loading a metadata snapshot, StandardAuthorizer must load all the records atomically as a group, rather than applying them one-by-one.

This is most important when a snapshot load happens in an already started process. One example of this is where a broker has fallen behind and needs to reload its state from a snapshot.

Initial Authorizer Setup

Authorizer#start returns some futures which should not be completed until the Authorizer is in a functional state. The intention here is to wait until we have some data that can be used for authorization before starting the SocketServer.

Note that on the broker, KIP-631 already specifies that we should not start the SocketServer until the controller has declared the broker "caught up." Therefore, even if the futures returned from StandardAuthorizer#start were completed immediately, there would be no harm done. However, on the controller, we must communicate with other controllers during the startup process, in order to establish a quorum. This requires that the authorizer be functional during that time.

In the case of StandardAuthorizer, we consider the Authorizer to be "ready to go" once it has loaded at least one metadata snapshot.

Compatibility, Deprecation, and Migration Plan

ZooKeeper-Based Clusters

This KIP has no impact on ZooKeeper-based clusters. The changes only apply to KRaft-based clusters.

If a user attempts to configure StandardAuthorizer when running in ZK mode, a fatal exception will be thrown indicating the problem.

New Default

For KRaft-based clusters, users that currently use AclAuthorizer can continue to do so. However, they will need to configure AclAuthorizer explicitly rather than getting it as a default.

Migration

If a user transitions to a different authorizer, the previous ACLs stored by StandardAuthorizer still remains stored in __cluster_metadata. They will not be deleted.

Note that we do not currently have any way of loading ACLs from ZooKeeper into this authorizer. However, we will design a mechanism for doing this in a future KIP where we define how to do migration from a ZooKeeper-based Kafka cluster to a KRaft-based Kafka cluster.

Rejected Alternatives

Leave AclAuthorizer as the Default

We could leave AclAuthorizer as the default for KRaft clusters. However, this would be highly confusing to new users, since it would introduce a ZooKeeper dependency when running in KRaft mode. Since KRaft is in Preview mode, we feel that now is the best time to change the default Authorizer for KRaft mode.