EnvelopeRequest handling

Master KIP

...

For older requests that need redirection, we shall create a new RPC called `Envelope` to embed the actual request. This request will be fully wrapping an older version client request, including its header, security information, actual data fields, etc. The request requires ClusterAction on CLUSTER.Forwarding broker will do the authentication first on the request. If the client request looks good, forwarding broker will redirect the incoming request, and let the controller to continue the security verification for consistency.

Public Interfaces

Protocol Bumps

...

We are also going to add a new RPC type to wrap the original request during the forwarding. We will make corresponding changes to `ApiMessageTypeGenerator` class to recognize the new field `Header` and `ApiMessage` during the auto generation. And for authentication and audit logging purpose, we proposed to add the following fields:

1. Serialized Principal information
2. Client host ip address
3. Listener name
4. Security protocol being used

...

When receiving an EnvelopeRequest, the broker shall authorize the request with forwarding broker's principal. If the outer request is verified, the broker will continue to unwrap the inner request and handle it as normal, which means it would continue performing authorization for the inner layer principal. For KIP-590 scope, the possible top error codes are:

...

The CLUSTER authorization for EnvelopeRequest takes place during the request handling, similar to LeaderAndIsrRequest. This ensures the EnvelopeRequest is not sent from a malicious client pretending to be a fellow broker. Once that part is done, the controller broker will not do a second authentication as it should already be done by the forwarding broker (consider a connecting flight, when entering the boarder of one country we always do the customs review in first hop). For inner request error, it will still be embedded inside the `ResponseData` struct defined in EnvelopeResponse below.

...

• We discussed about the possibility of immediately building a metadata topic to propagate the changes. This seems aligned with the eventual metadata quorum path, but at a cost of blocking the current API migration towards the bridge release, since the metadata quorum design is much more complicated and requires more iterations. To avoid this extra dependency on other tracks, we should go ahead and migrate existing protocols to meet the bridge release goal sooner.
• We discussed about adding a tag field to represent the original request principal name to the request header for controller audit log purpose. This approach potentially has security concern and making a principal field optional also has potential risk of getting ignored by older brokers when IBP is badly configured. So instead we choose the envelope approach which is more old-fashioned, standard and secure. Plus, upgrading the request to newer version in the middle of proxy could generate unexpected missing fields which are also tricky to be deal with each RPC bump case by case.
• We also discussed having the forwarding broker do the authentication and only forward valid resources admin requests. This approach adds up the complexity of doing audit logging as a de-centralized approach, and we decide not to perform such optimization and let controller do the raw request authentication and logging instead.

Future Works

We have also discussed about migrating the metadata read path to controller-only for read-after-write consistency. This sounds like a nice improvement but needs more discussions on trade-offs between overloading controller and the metadata consistency, also the progress of Raft quorum design as well.

...