...

We are also going to add a new RPC type to wrap the original request during the forwarding. We will make corresponding changes to `ApiMessageTypeGenerator` class to recognize the new field `Header` and `ApiMessage` during the auto generation. The PrincipalName and PrincipalType fields are for logging purpose.And for audit logging purpose, we proposed to add fields:

1. Serialized Principal information
2. Client host ip address
3. Listener name
4. Security protocol being used

{
  "apiKey": N,
  "type": "request",
  "name": "EnvelopeRequest",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "RequestHeader", "type": "Header", "versions": "0+",
      "about": "The embedded request header." },
	{ "name": "RequestData", "type": "ApiMessage", "versions": "0+",
      "about": "The embedded request data."},
	{ "name": "PrincipalTypePrincipalInfo", "type": "stringbytes", "versions": "0+",
      "about": "The embedded requestserialized principal typeinformation."},	
	    { "name": "PrincipalNameClientHostIP", "type": "string", "versions": "0+"}, 
     { "aboutname": "The embedded request principal name.ListenerName", "type": "string", "versions": "0+"},	
  ]
}

EnvelopeRequest Handling

  { "name": "SecurityProtocol", "type": "string", "versions": "0+"}
  ]
}

EnvelopeRequest Handling

When receiving an EnvelopeRequest, the broker shall authorize the When receiving an EnvelopeRequest, the broker shall authorize the request with forwarding broker's principal. If the outer request is verified, the broker will continue to unwrap the inner request and handle it as normal. For KIP-590 scope, the possible top error codes are:

...

For whatever result the controller replies to the inner request, the forwarding broker won't check. As long as the top level has no error, the forwarding broker will claim the request to be successful and reply the inner response to the admin client for the rest of error handling.

Monitoring Metrics

KafkaPrincipal Serialization

We shall also bring in a backward-incompatible change to add serializability to the KafkaPrincipalBuilder class:

public interface KafkaPrincipalBuilder {
    ...
	ByteBuffer serialize();

	KafkaPrincipal deserialize(ByteBuffer serializedPrincipal);
}

which requires a non-default implementation to ensure the principal information gets properly compacted into the forwarding request.

Monitoring Metrics

To effectively monitor the admin request forwarding To effectively monitor the admin request forwarding status, we would add two metrics: num-client-forwarding-to-controller-rate and num-client-fowarding-to-controller-count to visualize the progress. They are set up to track the number of clients being on a pending forward request for each broker.

...

As we discussed in the request routing section, to work with an older client, the first contacted broker need to act as a proxy to redirect the write request to the controller. To support the proxy of requests, we need to build a channel for brokers to talk directly to the controller. This part of the design is internal change only and won’t block the KIP progress.as a proxy to redirect the write request to the controller. To support the proxy of requests, we need to build a channel for brokers to talk directly to the controller. This part of the design is internal change only and won’t block the KIP progress.

Compatibility Breakage

The KafkaPrincipal builder serializability is a binary incompatible change stated in the KIP. For the smooth rollout of this KIP, we will defer this part of the implementation until we hit next major incompatible release, i.e Apache Kafka 3.0. This means we will breakdown the effort as:

1. For next 2.x release:
   1. Get new admin client forwarding changes
   2. Get the Envelope RPC implementation
   3. Get the forwarding path working and validate the function with fake principals in testing environment, without actual triggering in the production system
2. For next 3.0 release:
   1. Introduce serializability to PrincipalBuilder
   2. Turn on forwarding path in production and perform end-to-end testing

Rejected Alternatives

• We discussed about the possibility of immediately building a metadata topic to propagate the changes. This seems aligned with the eventual metadata quorum path, but at a cost of blocking the current API migration towards the bridge release, since the metadata quorum design is much more complicated and requires more iterations. To avoid this extra dependency on other tracks, we should go ahead and migrate existing protocols to meet the bridge release goal sooner.
• We discussed about adding a tag field to represent the original request principal name to the request header for controller audit log purpose. This approach potentially has security concern and making a principal field optional also has potential risk of getting ignored by older brokers when IBP is badly configured. So instead we choose the envelope approach which is more old-fashioned, standard and secure. Plus, upgrading the request to newer version in the middle of proxy could generate unexpected missing fields which are also tricky to be deal with each RPC bump case by case.

...