Page History

...

Change AlterConfig Request Routing

During the bridge release, the propagation path shall be revised as:

1. The admin client issues an (Incremental)AlterConfig request to the controller

2. The controller updates the config, and stores it in ZK

3. If #3 successful, returns to the client

4. ZK update will be propagated towards all affected brokers in the cluster

This The new simple routing change makes sure only controller needs to write to ZK, while other broker shall just wait for the metadata update from ZK notification eventually. As we have the source of truth configs stored in ZK still, any re-election of controller shall be safe.

The above path works with the new admin client we are going to introduce with next release. For older clients, they are not expected to forward the request to controller, so For admin RPCs who are currently sending directly to the controller, brokers should support the proxy of such requests, with a revised update path during the bridge release:

1. The old admin client issues an (Incremental)AlterConfig request to a random broker

2. The broker redirects the request to the controller

3. The controller updates the config, and store it in ZK

4. If #3 successful, returns to the proxy broker

5. The proxy broker returns to the client as success

6. ZK update will be propagated towards all affected brokers in the cluster

This whole update strategy change would be applied to all the direct ZK mutation paths, including:

...

In addition, to avoid exposing this forwarding power to the admin clients, the routing request shall be forwarded towards the controller broker internal endpoint which should be only visible to other brokers inside the cluster in the KIP-500 controller. Any admin configuration request with broker principal should not be going through the public endpoint and will be rejected for security purpose.

...

Starting from the first release version of KIP-590,  the following RPCs shall be forwarded to the controller from any broker:

• AlterPartitionReassignment
• CreatePartition
• CreateTopics
• DeleteTopics
• UpdateFeatures (ongoing with KIP-584)

And they would follow the same configuration request forwarding strategy discussed in the previous section.

The reason is that we shall remove "ControllerNodeProvider" on the admin client, so that clients no longer have direct access towards the controller. Thus the active controller is properly isolated from the outside world, according to the KIP-631.

Protocol Bump

We also need to bump the Metadata RPC to v10 to propagate internal topic creation policy violation. Specifically:

...

To support the authorization of RPCs during redirection, we would let CLUSTER_ACTION to override the following operation principals:

ALTER

ALTER

ALTER_CONFIGS

CREATE

token authentication

DELETE

This ensures that the forwarding broker could use its own principal to authenticate and proceed on certain ZK mutation operations. To distinguish which request is forwarded, the controller will try to differentiate requests coming from inter broker listener and advertised listener. If the request is from inter broker listener, we treat it as a forwarding request and do the override authentication.

...

Unfortunately for older admin clients they couldn't interpret this code, so an UNKNOWN_SERVER_ERROR will be presented, which is less ideal but still good enough to motivate users on checking to check the broker side log for authorization failure. We intended to avoid returning AUTHORIZATION failure to the old client so that users don't waste time debugging any client side security setup.

...