Page History

...

Current state: Accepted

Discussion thread: here

JIRAJIRAs:

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

To be more strict of protecting controller information, the "ControllerId" field in new MetadataResponse shall be set to a random broker for v0-v9 request, and gets deprecated on v10. Note that only existing clients are using Metadata RPC to get controller info, so it should be safe to deprecate and we would explicitly mention that on the NetworkClient meta comments.

Binding with IBP (inter.broker.protocol)

To make sure the admin client request version could be properly processed by both the forwarding broker and the controller broker, we need to treat the zk mutation RPCs as inter-broker communication, which means we need to guard those protocol bump behind IBP. Starting from the first version introducing redirection, all affected RPCs have to bump IBP when they are bumping their protocol again, so that broker could return accurate version range inside ApiVersionResponse.

New Envelope RPC

We are also going to add a new RPC type to wrap the original request during the forwarding. We will make corresponding changes to `ApiMessageTypeGenerator` class to recognize the new field `Header` and `ApiMessage` during the auto generation. And for authentication and audit logging purpose, we proposed to add the following fields:

1. Serialized Request Data
2. Id token for authentication purpose
   1. Request principal for authentication and authorization purpose
3. Client hostname for authentication purpose 

New Envelope RPC

We are also going to add a new RPC type to wrap the original request during the forwarding. We will make corresponding changes to `ApiMessageTypeGenerator` class to recognize the new field `Header` and `ApiMessage` during the auto generation. And for authentication and audit logging purpose, we proposed to add the following fields:

1. Serialized Request Data
2. Id token for authentication purpose
   1. Request principal for authentication and authorization purpose
3. Client hostname for authentication purpose 

{
  "apiKey": N,
  "type": "request",
  "name": "EnvelopeRequest",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "RequestData", "type": "bytes", "versions": "0+", "zeroCopy": true,
      "

{
  "apiKey": N,
  "type": "request",
  "name": "EnvelopeRequest",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "RequestData", "type": "bytes", "versions": "0+", "zeroCopy": true,
      "about": "The embedded request header and data."},
    { "name": "PrincipalIdToken", "type": "bytes", "tag": 0, "taggedVersions": "0+" },
    { "name": "RequestPrincipal", "type": "bytes", "versions": "0+", "zeroCopy": true,
      "ignorable": true, "nullableVersions": "0+", "default": "null",
      "about": "Value of the initial client principal when the request is redirected by a broker." },
    { "name": "ClientHostName", "type": "string", "versions": "0+", "default": "",
      "about": "The original client's hostname." }
  ]
}

...

An Authorizer can reject impersonated requests by checking if the forwarding principal is present. This information is obviously useful for auditing as wellby checking if the forwarding principal is present. This information is obviously useful for auditing as well.

ApiVersion Consistency

Admin clients send ApiVersions to the broker upon the first connection establishes. The tricky thing after forwarding is enabled is that for forwardable APIs, admin client needs to know a commonly-agreed rang of ApiVersions among handling broker, active controller and itself.

Right now the inter-broker APIs are guaranteed by IBP constraints, but not for forwardable APIs. A compromised solution would be to put all forwardable APIs under IBP, which is brittle and hard to maintain consistency.

Instead, any broker connecting to the active controller should send an ApiVersion request from beginning, so it is easy to compute that information and send back to the admin clients upon ApiVersion request from admin.  Any rolling of the active controller will trigger reconnection between broker and controller, which guarantees a refreshed ApiVersions between the two. This approach avoids the tight bond with IBP and broker could just close the connection between admin client to trigger retry logic and refreshing of the ApiVersions. Since this failure should be rare, two round-trips and timeout delays are well compensated by the less engineering work.

Routing in KIP-500 

In addition, to avoid exposing this forwarding power to the admin clients, the routing request shall be forwarded towards the controller broker internal endpoint which should be only visible to other brokers inside the cluster in the KIP-500 controller. Any admin configuration request with broker principal should not be going through the public endpoint and will be rejected for security purpose. For pre-KIP-500 controller, we would allow broker principal to go through only when the message comes in on the inter-broker listener, which is an indication of a forwarding request. The pre-KIP-500 cluster could not fully prevent malicious client pretending to be a forwarding request, but the attacker must have super user access to gain CLUSTER_ACTION.

...