...

Here we use ZK as the persistent storage for all the config changes, and even some brokers are not able to get in sync with ZK due to transient failures, a successful update shall be eventually guaranteed. In this KIP we would like to maintain the same level of guarantee, and make the controller as the single writer to modify the config metadata in ZK.

Proposed

...

ChangesExisting RPCs which are sending 

Change How ZK Mutation Request Routs

...

It is because we are trying to isolate controller access from the admin client in the post KIP-500 world. Existing RPCs which are sending Old admin clients who are sending requests directly to the controller will be given a random broker id and rely on forwarding of the original requests as well.

Internal CreateTopicsRequest Routing 

...

For ZK mutation requests that need redirection, forwarding broker will just use its own authorizer to verify the principals. When the request looks good, it will just forward the request with its own credentials, so that the controller broker will only validate the broker principal in the forwarded request. The only exceptional case is the controller audit log which needs a principal name of the request, so we will add an optional tag called "InitialPrincipalName" to the header when sending the proxy request.

To better understand how security check works, take AlterConfig for an example, the intended workflow for a KIP-590 broker would be:

Step 1. Filter out resources that are authorized
         1.1 Use traditional principals to verify first. If authorized, continue
         1.2 If not authorized, check whether the request is from the control plane. Note that this is a best-effort to verify whether the request is internal.
         1.3 If the request is not from the control plane, return authorization failure
         1.4 If the request is from the control plane, use CLUSTER_ACTION to verify and determine the result

Step 2. Check the request context to see if this is a forwarding request, by checking whether it is from control plane and uses extra header fields
        2.1 if the resource is authorized, and if this is the active controller, process it
        2.2 if the resource is authorized but this is not the active controller, return NOT_CONTROLLER to the sender (forwarding broker) for retry
        2.3 if the resource is not authorized, return CLUSTER_AUTHORIZATION_FAILURE to propagate back to the original client through forwarding broker

Step 3. If the request is not a forwarding request
        3.1 If the resource is authorized, and this is the active controller, process it
        3.2 If the resource is authorized, but this is not active controller, put the resource into the preparation for a new AlterConfig request for forwarding
        3.3 If the resource is not authorized, reply the original client AUTHORIZATION_FAILURE when the forwarding request is returned

In addition, to avoid exposing this forwarding In addition, to avoid exposing this forwarding power to the admin clients, the routing request shall be forwarded towards the controller broker internal endpoint which should be only visible to other brokers inside the cluster in the KIP-500 controller. Any admin configuration request with broker principal should not be going through the public endpoint and will be rejected for security purpose. For pre-KIP-500 controller, we would allow broker principal to go through only when the message comes in on the inter-broker listener, which is an indication of a forwarding requestlistener, which is an indication of a forwarding request. The pre-KIP-500 cluster could not fully prevent malicious client pretending to be a forwarding request, but the attacker must have super user access to gain CLUSTER_ACTION.

Public Interfaces

Deprecate Client Side Controller Access

...

{
  "type": "header",
  "name": "RequestHeader",
  // Version 0 of the RequestHeader is only used by v0 of ControlledShutdownRequest.
  //
  // Version 1 is the first version with ClientId.
  //
  // Version 2 is the first flexible version.
  "validVersions": "0-2",
  "flexibleVersions": "2+",
  "fields": [
    { "name": "RequestApiKey", "type": "int16", "versions": "0+",
      "about": "The API key of this request." },
    { "name": "RequestApiVersion", "type": "int16", "versions": "0+",
      "about": "The API version of this request." },
    { "name": "CorrelationId", "type": "int32", "versions": "0+",
      "about": "The correlation ID of this request." },
    ...
    // ----- new optional field ----
    { "name": "InitialPrincipalName", "type": "string", "tag": 0, "taggedVersions": "2+", "ignorable": true,
      "about": "Optional value of the initial principal name when the request is redirected by a broker." },
    { "name": "InitialClientId", "type": "string", "tag": 0, "taggedVersions": "2+", "ignorable": true,
      "about": "Optional value of the initial client id when the request is redirected by a broker." },
    // ----- end new field ---------
  ]
}

For audit logging purpose, we added the initial principal name. 

For KIP-599 throttling requirement, both the initial principal name and the initial client id are required to be presentThe purpose of adding principal name is for the audit logging, and the client id is being used to throttling according to KIP-599 requirement.

Monitoring Metrics

To effectively monitor the admin request forwarding status, we would the following metered metric:

...

• We discussed about the possibility of immediately building a metadata topic to propagate the changes. This seems aligned with the eventual metadata quorum path, but at a cost of blocking the current API migration towards the bridge release, since the metadata quorum design is much more complicated and requires more iterations. To avoid this extra dependency on other tracks, we should go ahead and migrate existing protocols to meet the bridge release goal sooner.

• We thought about adding an alerting metrics called request-forwarding-to-controller-authorization-fail-count in an effort to help administrator detect wrong security setup sooner. However, there should already be metrics monitoring request failures, so this metric could be optional.

• We thought about monitoring older client connections in the long term after bridge release, when we perform some incompatible changes to the Raft Quorum, to better capture the timing for a major version bump. However, KIP-511 also has already exposed metrics like an "unknown" software name and an "unknown" software version which could serve for this purpose.

• We discussed about adding a new RPC type called Envelope to wrap the original request during the forwarding. Although the Envelope API provides certain privileges like data embedding and principal embedding, it creates a security hole by letting a malicious user impersonate any resending broker. Passing the principal around also increases the vulnerability, compared with other standard ways such as passing a verified token, but it is unfortunately not fully supported with Kafka security. So for the security concerns, we are abandoning the Envelope approach and fallback to just forward the raw admin requests.
• We discussed about maintaining the client access to the controller, which has conflicts with KIP-631, so we decide to go extra steps to give existing ZK mutation RPC with forwarding power as well.

Future Works

We have also discussed about migrating the metadata read path to controller-only for read-after-write consistency. This sounds like a nice improvement but needs more discussions on trade-offs between overloading controller and the metadata consistency, also the progress of Raft quorum design as well.

...