Status

Current state: Under Discussion Approved

Discussion thread:   DISCUSS DISCUSS+VOTE

JIRA:   KAFKA-13646

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

Currently, when using KRaft mode, users still have to have an Apache ZooKeeper instance if they want to use AclAuthorizer. We should have a built-in Authorizer for KRaft mode that does not depend on ZooKeeper. This KIP introduces such an authorizer, called StandardAuthorizer.

...

Design

...

Overview

This KIP creates org.apache.kafka.metadata.authorizer.StandardAuthorizer

...

The StandardAuthorizer is , a new Authorizer class which stores its ACLs in the __cluster_metadata topic. It is used by default in KRaft clusters if the , unless the administrator configures a different Authorizer by setting authorizer.class.name configuration is not set. Note that the default for ZooKeeper-based clusters remains the existing AclAuthorizer class.

In general, StandardAuthorizer should act acts as a drop-in replacement for AclAuthorizer, implementing . It implements all the same behaviors. It will also support supports the same static configuration keys as AclAuthorizer, including super.users and allow.everyone.if.no.acl.found.

Consistency

StandardAuthorizer stores ACLs in the __cluster_metadata topic. As described in KIP-500 and KIP-631, brokers and standby controllers continuously read this log, up to its last stable offset. This means that at any given time, they will have an Authorizer state corresponding to some point on a single timeline. This is similar to the consistency guarantees we provide today with the ZooKeeper-based AclAuthorizer.

Just as with all other metadata state, the authorization state on the active controller may be slightly ahead of the rest of the cluster. If the active controller succeeds in persisting this new state to the log (by advancing the last stable offset) all other nodes will pick it up. Otherwise, if the active controller fails, a new controller will take over and resume from the last stable offset. See KIP-631 for more information.

Ordering

ACL records must be applied in order, even if they appear in the same batch of records. This is important because StandardAuthorizer, like other Authorizers, is multi-threaded and will continue to authorize new operations even while record changes are applied.

For example, if we have two ACL records like this:

1. Deny user bob access to topic foo
2. Allow user bob access to all topics

We know that when both an ALLOW and a DENY ACL apply to the same access attempt, access must be denied. Therefore, the net effect of applying #1 and then #2 is to give bob access to all topics except for foo. However, if we apply record #2 before record #1, this will result in a brief window of time during which user bob incorrectly has access to topic foo. We must avoid such situations by always applying ACL records in the order they appear in the __cluster_metadata log.

Initialization

In general, Authorizer implementations are expected to block for a certain amount of time when they start up, while they are loading their ACLs. It would be inappropriate to start authorizing requests immediately, even before this ACL data was present. To see why, consider the situation where allow.everyone.if.no.acl.found is set to true. When this configuration is set, and we have not yet loaded any ACLs, anyone could access the system. Clearly, having a brief period during startup when anyone can access the system is not acceptable.

Of course, in a system based on streaming metadata changes, there is no "final change" to apply, no terminal state. However, by waiting to start the authorizer until we have replayed ACL records up to the high water mark, we can at least ensure that we don't expose an authorizer state from the past to users during broker and controller startup.

Similarly, when a node has fallen behind and must apply a snapshot, that snapshot should be applied atomically.

Bootstrapping

When we have just created a new cluster, there are no ACL records present yet. This presents a problem: the nodes must communicate with the controller quorum, but there are no ACLs which could allow them to do that. We can solve this problem in one of two ways: by setting set of users that have all permissions) and allow.everyone.if.no.acl.found to true, or by putting the user(s) for the brokers and controllers into super.users. Since most system administrators prefer to operate in a "deny by default" regime, we expect the second solution to be used most often.

In the future, we plan on making more sophisticated KRaft bootstrapping options available – for example, commands that could set up the local metadata log on each node during installation. This will also be useful for things like dynamically configured SCRAM setups, and so forth. However, the simple solution described above should suffice for now.

Early Start Listeners

We know that any user that appears in super.users will always be authorized, even if the StandardAuthorizer loading process is not yet complete. This lets us implement a useful optimization: if a listener appears in early.start.listeners, we will immediately start that listener, even before StandardAuthorizer has completed loading up to the __cluster_metadata high water mark. However, until loading has fully completed, we will only allow super users to send requests to this endpoint. All other users will get AUTHORIZER_NOT_READY.

Public Interfaces

Records

AccessControlEntryRecord

...

{
  "apiKey": ...,
  "type": "metadata",
  "name": "RemoveAccessControlEntryRecord",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "Id", "type": "uuid", "versions": "0+",
      "about": "The ID of the AccessControlEntry to remove." }
  ]
}

Metadata Shell

The metadata shell will support examining KRaft ACLs. Each ACL will appear in /acl/id/<uuid> in its JSON form.

Design

Contexts

StandardAuthorizer runs in two contexts: on the broker, and in the controller. Just like currently, a node running in combined mode (broker+controller) will have two separate Authorizer instances: one for the broker, and one for the controller.

In most cases, these Authorizer objects should have the same state. However, when the controller is active, its Authorizer state may be slightly ahead of the the broker's Authorizer state. This will happen in the time between when a new ACL is created (or deleted) and the time that this change is persisted durably by a majority of controller quorum peers.

Atomic Snapshot Load

Unlike many other Kafka components, the Authorizer is accessed from many different threads without locking. Any change to the Authorizer state will be reflected immediately in those threads. Therefore, when loading a metadata snapshot, StandardAuthorizer must load all the records atomically as a group, rather than applying them one-by-one.

This is most important when a snapshot load happens in an already started process. One example of this is where a broker has fallen behind and needs to reload its state from a snapshot.

Initial Authorizer Setup

Authorizer#start returns some futures which should not be completed until the Authorizer is in a functional state. The intention here is to wait until we have some data that can be used for authorization before starting the SocketServer.

Note that on the broker, KIP-631 already specifies that we should not start the SocketServer until the controller has declared the broker "caught up." Therefore, even if the futures returned from StandardAuthorizer#start were completed immediately, there would be no harm done. However, on the controller, we must communicate with other controllers during the startup process, in order to establish a quorum. This requires that the authorizer be functional during that time.

Configuration Keys

EnvelopeRequest

We will bump the version of EnvelopeRequest to reflect the fact that it can now return a new error code: AUTHORIZER_NOT_READY. This error code will only ever be returned from early start endpoints. It indicates that the operation could not be performed because the Authorizer has not fully initialized yet, as described in the previous section about "early start listeners."

As described in KIP-590, brokers use EnvelopeRequest to forward user requests to KRaft controllers. When the broker gets back an AUTHORIZER_NOT_READY error for a forwarded request, it should wait for a while and then try to forward the request again. This is basically the same behavior as the what the broker does when there is a network error when forwarding a request. At some point, of course, the request will time out.

If the version of EnvelopeRequest is too old, then the controller will return UNKNOWN_SERVER_EXCEPTION instead.

Since the controller uses ApiVersions to determine what RPC versions to use, rather than consulting the IBP, we do not need to bump the IBP to make this change.

Metadata Shell

The metadata shell will support examining KRaft ACLs. Each ACL will appear in /acl/id/<uuid> in its JSON form.

New Metrics

In order to improve manageability, we will add a new metric, AclCount.

New Authorizer Function

 In order to support the AclCount metric, we will extend the Authorizer API with a new aclCount function.

int aclCount()

In order to preserve compatibility, this function will default to returning -1, so that existing Authorizer subclasses will continue to work. Authorizers that expose this metric should override this function with the correct valueIn the case of StandardAuthorizer, we consider the Authorizer to be "ready to go" once it has loaded at least one metadata snapshot.

Compatibility, Deprecation, and Migration Plan

...