Page History

...

We are also going to add a new RPC type to wrap the original request during the forwarding. We will make corresponding changes to `ApiMessageTypeGenerator` class to recognize the new field `Header` and `ApiMessage` during the auto generation. And for authentication and audit logging purpose, we proposed to add the following fields:

1. Serialized Request Data
2. Initial Principal Name for audit logging and throttling purpose
3. Id token for authentication purpose
   1. Request principal for authentication and authorization purpose
4. Client hostname for authentication Initial Client Id for throttling purpose 

{
  "apiKey": N,
  "type": "request",
  "name": "EnvelopeRequest",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    {     { "name":  "RequestData",  "type":  "ApiMessagebytes",  "versions":  "0+",
       "zeroCopy": true,
      "about":  "The embedded request header and data."},
      { "name": "InitialPrincipalNamePrincipalIdToken", "type": "stringbytes", "ignorabletag": true0,
      "abouttaggedVersions": "Optional value of the initial principal name when the request is redirected by a broker.0+" },
    { "name": "InitialClientIdRequestPrincipal", "type": "stringbytes", "ignorableversions": true,"0+", "zeroCopy": true,
      "ignorable": true, "nullableVersions": "0+", "default": "null",
      "about": "Optional valueValue of the initial client idprincipal when the request is redirected by a broker." },
  ]
}    { "name": "ClientHostName", "type": "string", "versions": "0+", "default": "",
      "about": "The original client's hostname." }
  ]
}

When receiving an EnvelopeRequest, the broker shall authorize the request with forwarding broker's principal. If the outer request is verified, the broker will continue to unwrap the inner request and handle it as normal, which means it would continue performing authorization for the inner layer principal. For KIP-590 scope, the When receiving an EnvelopeRequest, the broker shall authorize the request with forwarding broker's principal. If the outer request is verified, the broker will continue to unwrap the inner request and handle it as normal, which means it would continue performing authorization for the inner layer principal. For KIP-590 scope, the possible top error codes are:

...

{
  // Possible top level error code:
  //
  // NOT_CONTROLLER
  // CLUSTER_AUTHORIZATION_FAILED
  //
  "apiKey": N,
  "type": "response",
  "name": "EnvelopeResponse",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    {     { "name":  "ResponseDataThrottleTimeMs",  "type":  "ApiMessageint32",  "versions":  "0+",
            "about":  "The embedded response data."},
    { "name": "ErrorCode", "type": "int16", "versions":  duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
    { "name": "ResponseData", "type": "bytes", "versions": "0+",
       "aboutnullableVersions": "The error code, or 0 if there was no error." },
  ]
}

EnvelopeResponse Handling

 "0+",
      "zeroCopy": true, "default": "null",
      "about": "The embedded response header and data."},
    { "name": "ErrorCode", "type": "int16", "versions": "0+",
      "about": "The error code, or 0 if there was no error." }
  ]
}

EnvelopeResponse Handling

When the response contains NOT_CONTROLLER error code, the forwarding broker will keep finding the When the response contains NOT_CONTROLLER error code, the forwarding broker will keep finding the correct controller until request eventually times out. For CLUSTER_AUTHORIZATION_FAILED, this indicates an internal error for broker security setup which has nothing to do with the client, so we have no other way but returning an UNKNOWN_SERVER_ERROR to the admin client. 

...

Although some users may configure the same listener name for both client and inter broker communication, which invalidates the differentiation process, this override approach still guarantees no extra security access breach since CLUSTER_ACTION implies either the broker or a super user.

Principal Serialization

In Kafka, principals are represented by the KafkaPrincipal type. Users are allowed to provide their own extensions through the use of a KafkaPrincipalBuilder. Extensions may include additional fields (such as a tenant ID), so we need a new mechanism to serialize and deserialize a principal. For this, we will use a new KafkaPrincipalSerde type:

interface KafkaPrincipalSerde {
  byte[] serialize(KafkaPrincipal principal);
  KafkaPrincipal deserialize(byte[] bytes);
}

Until 3.0, this type will be optional. If it is not implemented by the KafkaPrincipalBuilder type, then the broker will log a warning when the broker starts up, thus disabling the redirection as well. After 3.0, it will be required and the broker will not start without it.

Some users may not want to allow impersonation of some APIs even going beyond the limited set of supported APIs. For example, a user might prefer to allow ACL changes only from within a private network. For this use case, we extend the Authorizer so that it can distinguish impersonated requests. Specifically, we propose to add the principal that is forwarding the request to be included in the authorization context:

public interface AuthorizableRequestContext {
  Optional<KafkaPrincipal> forwardingPrincipal();
}

An Authorizere can reject impersonated requests by checking if the forwarding principal is present. This information is obviously useful for auditing as well.

Routing in KIP-500 

In addition, to avoid exposing this forwarding power to the admin clients, the routing request shall be forwarded towards the controller broker internal endpoint which should be only visible to other brokers inside the cluster in the KIP-500 controller. Any admin configuration request with broker principal should not be going through the public endpoint and will be rejected for security purpose. For pre-KIP-500 controller, we would allow broker principal to go through only when the message comes in on the inter-broker listener, which is an indication of a forwarding request. The pre-KIP-500 cluster could not fully prevent malicious client pretending to be a forwarding request, but the attacker must have super user access to gain CLUSTER_ACTION.

...