...

Proposed Changes

NetworkClient

1. The new config will be a common client config. The NetworkClient will keep the proposed configs as new properties.
2. NetworkClient.poll() will iterate all connecting nodes and disconnect those timed out connections using the exact approach as it handles “request.timeout.ms” 
3. The node providing criteria C in the leastLoadedNode() will also change accordingly. Now the criteria should look like below:
   1. Provide the connected node with least number of inflight requests
   2. If no connected node exists, provide the connecting node with the largest index in the cached list of nodes.
   3. If no connected or connecting node exists, provide the disconnected node which respects the reconnect backoff and is least recently provided. Consider the case when we have multiple DISCONNECTED nodes and the time interval between the two provide() invokes is greater than reconnect.backoff.ms. The Provider can provide the same nodes all the time. Thus, the provider should provide the least recently provided nodes among all nodes passing the canConnect() check.

ClusterConnectionStates 

1. Add a new HashSet property ConnectingNodes keeping all the connecting node ids.
2. Will expose a public API that returns the ConnectingNodes mentioned in #1, helping the NetworkClient process the timeout iteration.
3. State transition:
   1. ClusterConnectionStates.connecting() will add the node id to ConnectingNodes
   2. ClusterConnectionStates.ready() will remove the node id to ConnectingNodes
   3. ClusterConnectionStates.disconnected() will remove the node id from ConnectingNodes

When would the connection timeout increase?

Every time the timeout hits, the timeout value of the next connection try will increase. The timeout will hit iff a connection stays at the `connecting` state longer than the timeout value, as indicated by ClusterConnectionStates.NodeConnectionState. The connection state of a node will change iff SelectionKey.OP_CONNECT is detected by nioSelector.Select(). The connection state may transit from `connecting` to 

...

In other words, the timeout will hit and increase iff the interested SelectionKey.OP_CONNECT doesn't happen before the timeout arrives, which means, for example, network congestion, failure of the ARP request, packet filtering, routing error, or a silent discard may happen.

Relationship between the proposed connection timeout, existing request timeout, and existing API timeout

Connection timeout dominates request timeout and API timeout

When connection timeout hits, the connection will be closed. The client will be notified either by the responses constructed by NetworkClient or the callbacks attached to the request. As a result, the request failure will be handled before either connection timeout or API timeout arrives.

Neither request timeout or API timeout dominates connection timeout

Request timeout: Because request timeout only affects in-flight requests, after the API NetworkClient.ready() is invoked, the connection won't get closed after "request.timeout.ms” hits. Before

1. the SocketChannel is connected
2. SSL handshake finished
3. authentication has finished (SASL)

, clients won't be able to invoke NetworkClient.send() to send any request, which means no in-flight request targeting to the connection will be added.

API timeout: In AdminClient, API timeout acts by putting a smaller and smaller timeout value to the chain of requests in a same API. After the API timeout hits, the retry logic won't close any connection. In consumer, API timeout acts as a whole by putting a limit to the code block executing time. The retry logic won't close any connection as well.

Compatibility, Deprecation, and Migration Plan

...