...

For older requests that need redirection, we shall create a new RPC called `Envelope` to embed the actual request. This request will be fully wrapping an older version client request, including its header, security information, actual data fields, etc. The request requires ClusterAction on CLUSTER.

Forwarding broker will blindly redirect the incoming request, and let the controller to continue the security verification for consistency forwarding broker will just use its own authorizer to verify the principals. When the request looks good, it will just forward the request with its own credentials, no second validation needed. The only exceptional case is the controller audit log which needs a principal name of the request, so we will add an optional tag called "PrincipalName" to the header when sending the proxy request.

Public Interfaces

Protocol Bumps

We are going to bump all mentioned mutation APIs above by one version, and new admin client was expected to only talk to the controller. For example we bump the AlterConfig API to v2.

...

If the request is v1, broker with new AlterConfig protocol enabled should always proxy the request to the controller by sending a v2 request. If  If the request is v2, the recipient broker should handle it or respond a NOT_CONTROLLER to ask for rediscovery if it is not the controller.This effectively means the request shall never be forwarded twice after bumping from v1 to v2, which avoids the edge scenario for infinite redirection during unstable controller elections.

Same applies for all other mentioned requests as well. The new version request should always go to the controller. If the request is on an older version, broker shall redirect it to the controller.

...

The CreateTopic routing change is purely inter-broker. Since the CreateTopicRequest is already handled by controller only, so no change on this side.

New

...

Envelope RPC

We are also going to add a tag field to represent new RPC type to wrap the original request principal name to the request header for controller audit log purposeduring the forwarding. We will make corresponding changes to `ApiMessageTypeGenerator` class to recognize the new field `Header` and `ApiMessage` during the auto generation.

{
  "apiKey": N,
  "type": "headerrequest",
  "name": "RequestHeaderEnvelopeRequest",
   // Version 0 of the RequestHeader is only used by v0 of ControlledShutdownRequest.
  //
  // Version 1 is the first version with ClientId.
  //
  // Version 2 is the first flexible version.
  "validVersions": "0-2",
  "flexibleVersions": "2+",
  "fields": [
    "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "RequestHeader", "type": "Header", "versions": "0+",
      "about": "The embedded request header." },
	{ "name": "RequestData", "type": "ApiMessage", "versions": "0+",
      "about": "The embedded request data."},
	{ "name": "PrincipalType", "type": "string", "versions": "0+",
      "about": "The embedded request principal type."},
	{ "name": "RequestApiKeyPrincipalName", "type": "int16string", "versions": "0+",
      "about": "The embedded APIrequest key of this request." },
  principal name."},	
  ]
}

EnvelopeRequest Handling

When receiving an EnvelopeRequest, the broker shall authorize the request with forwarding broker's principal. If the outer request is verified, the broker will continue to unwrap the inner request and handle it as normal. For KIP-590 scope, the possible top error codes are:

• NOT_CONTROLLER as we are only forwarding admin write requests.
• AUTHORIZATION_FAILED if the inter-broker verification failed.

For inner request error, it will still be embedded inside the EnvelopeResponse below.

{
  // Possible top level error code:
  //
  // NOT_CONTROLLER
  // CLUSTER_AUTHORIZATION_FAILED
  //
  "apiKey": N,
  "type": "response",
  "name": "EnvelopeResponse",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "RequestApiVersionEmbeddedHeader", "type": "int16Header", "versions": "0+",
      "about": "The APIembedded version of this requestresponse header." },
    { "name": "CorrelationIdEmbeddedApiKey", "type": "int32int16", "versions": "0+",
      "about": "The correlationembedded ID of this requestAPI key." },
	{ "name": "EmbeddedResponseData", "type":  ..."ApiMessage", "versions": "0+",
    // ----- new optional field ----  "about": "The embedded response data."},
    { "name": "PrincipalNameErrorCode", "type": "stringint16", "tagversions": 0, "taggedVersions": "2"0+", "ignorable": true,
      "about": "OptionalThe valueerror ofcode, theor principal0 nameif whenthere thewas request is redirected by a brokerno error." }, 
    // ----- end new field ---------
  ]
}

EnvelopeResponse Handling

When the response contains NOT_CONTROLLER error code, the forwarding broker will keep finding the correct controller until request eventually times out. For CLUSTER_AUTHORIZATION_FAILED, this indicates an internal error for broker security setup which has nothing to do with the client, so we have no other way but returning an UNKNOWN_SERVER_ERROR to the admin client. 

For whatever result the controller replies to the inner request, the forwarding broker won't check. As long as the top level has no error, the forwarding broker will claim the request to be successful and reply the inner response to the admin client for the rest of error handling.

New Metrics

To effectively monitor the request forwarding status, we would add two metrics: request-forwarding-to-controller-rate and request-fowarding-to-controller-count to visualize the progress. .

We shall also add an alerting metrics called request-forwarding-to-controller-authorization-fail-count in an effort to help administrator detect wrong security setup sooner.

In the long term after bridge release, we could potentially perform some incompatible changes to the Raft Quorum. To monitor the old client connections to better capture the timing for a major version bump, we shall also expose a metric called old-client-connections-count beyond KIP-511. This metric will record those clients whose version is older than the first KIP-511 supported client, as they have no self identity to propose.

...

As we discussed in the request routing section, to work with an older client, the first contacted broker need to act as a proxy to redirect the write request to the controller. To support the proxy of requests, we need to build a channel for brokers to talk directly to the controller. This part of the design is internal change only and won’t block the KIP progress.

Rejected Alternatives

• We discussed about the possibility of immediately building a metadata topic to propagate the changes. This seems aligned with the eventual metadata quorum path, but at a cost of blocking the current API migration towards the bridge release, since the metadata quorum design is much more complicated and requires more iterations. To avoid this extra dependency on other tracks, we should go ahead and migrate existing protocols to meet the bridge release goal sooner.
• We discussed about adding a tag field to represent the original request principal name to the request header for controller audit log purpose. This approach potentially has security concern and making a principal field optional also has potential risk of getting ignored by older brokers when IBP is badly configured. So instead we choose the envelope approach which is more old-fashioned, standard and secure. Plus, upgrading the request to newer version in the middle of proxy could generate unexpected missing fields which are also tricky to be deal with each RPC bump case by case.

Future Works

We have also discussed about migrating the metadata read path to controller-only for read-after-write consistency. This sounds like a nice improvement but needs more discussions on trade-offs between overloading controller and the metadata consistency, also the progress of Raft quorum design as well.

...