...

• AlterConfig

• IncrementalAlterConfig 

• CreateAcls

• DeleteAcls

• AlterClientQuotas

• CreateDelegationToken

• RenewDelegationToken

• ExpireDelegationToken

Furthermore, starting from the first release version of KIP-590,  the following RPCs shall be forwarded to the controller from any broker as well:

• AlterPartitionReassignment
• CreatePartition
• CreateTopics
• DeleteTopics
• UpdateFeatures (ongoing with KIP-584)

It is because we are trying to isolate controller access from the admin client in the post KIP-500 world. Existing RPCs which are sending directly to the controller will rely on forwarding as well.

Internal CreateTopicsRequest Routing 

...

Deprecate Client Side Controller Access 

Starting from the first release version of KIP-590,  the following RPCs shall be forwarded to the controller from any broker:

• AlterPartitionReassignment
• CreatePartition
• CreateTopics
• DeleteTopics
• UpdateFeatures (ongoing with KIP-584)

And they would follow the same configuration request forwarding strategy discussed in the previous section.

Access

We The reason is that we shall remove "ControllerNodeProvider" on the admin client, so that new clients no longer have direct access towards the controller. All admin requests would try to use the same "LeastLoadedNodeProvider" to get a random node to talk to. Thus the active controller is properly isolated from the outside world, according to the KIP-631. 

...

Unfortunately for older admin clients they couldn't interpret this code, so an UNKNOWN_SERVER_ERROR will be presented, which is less ideal but still good enough to motivate users to check the broker side log for authorization failure. We intended to avoid returning AUTHORIZATION failure to the old client so that users don't waste time debugging any client side security setup.

New Tag for Principal Name And Client Id

We are also going to add a new tag field fields to represent the original request principal name and client id to the request header for controller audit log purpose.

{
  "type": "header",
  "name": "RequestHeader",
  // Version 0 of the RequestHeader is only used by v0 of ControlledShutdownRequest.
  //
  // Version 1 is the first version with ClientId.
  //
  // Version 2 is the first flexible version.
  "validVersions": "0-2",
  "flexibleVersions": "2+",
  "fields": [
    { "name": "RequestApiKey", "type": "int16", "versions": "0+",
      "about": "The API key of this request." },
    { "name": "RequestApiVersion", "type": "int16", "versions": "0+",
      "about": "The API version of this request." },
    { "name": "CorrelationId", "type": "int32", "versions": "0+",
      "about": "The correlation ID of this request." },
    ...
    // ----- new optional field ----
    { "name": "InitialPrincipalName", "type": "string", "tag": 0, "taggedVersions": "2+", "ignorable": true,
      "about": "Optional value of the initial principal name when the request is redirected by a broker." },
    { "name": "InitialClientId", "type": "string", "tag": 0, "taggedVersions": "2+", "ignorable": true,
      "about": "Optional value of the initial client id when the request is redirected by a broker." },
    // ----- end new field ---------
  ]
}

The purpose of adding principal name is for the audit logging, and the client id is being used to throttling according to KIP-599 requirement.

Monitoring Metrics

To effectively monitor the admin request forwarding status, we would the following metered metric:

...

The upgrade path shall be guarded by the inter.broker.protocol (IBP) to make sure the routing behavior is consistent. After first rolling bounce to upgrade the binary version, all fellow brokers are still handling ZK mutation requests by themselves. With the second IBP bump rolling bounce, all upgraded brokers will be using the new routing algorithm effectively described in this KIP. 

For older admin clients which still "try to" send the request to the controller, receiving broker will redirect the request to the active controller as stated in the KIP. The handling should be exactly the same no matter the ZK mutation request is from an old or a new admin client.

Rejected Alternatives

• We discussed about the possibility of immediately building a metadata topic to propagate the changes. This seems aligned with the eventual metadata quorum path, but at a cost of blocking the current API migration towards the bridge release, since the metadata quorum design is much more complicated and requires more iterations. To avoid this extra dependency on other tracks, we should go ahead and migrate existing protocols to meet the bridge release goal sooner.

• We thought about adding an alerting metrics called request-forwarding-to-controller-authorization-fail-count in an effort to help administrator detect wrong security setup sooner. However, there should already be metrics monitoring request failures, so this metric could be optional.

• We thought about monitoring older client connections in the long term after bridge release, when we perform some incompatible changes to the Raft Quorum, to better capture the timing for a major version bump. However, KIP-511 also has already exposed metrics like an "unknown" software name and an "unknown" software version which could serve for this purpose.

• We discussed about adding a new RPC type called Envelope to wrap the original request during the forwarding. Although the Envelope API provides certain privileges like data embedding and principal embedding, it creates a security hole by letting a malicious user impersonate any resending broker. Passing the principal around also increases the vulnerability, compared with other standard ways such as passing a verified token, but it is unfortunately not fully supported with Kafka security. So for the security concerns, we are abandoning the Envelope approach and fallback to just forward the raw admin requests.

...