Status

Current state: Discuss

Discussion thread: here

JIRA: Unable to render Jira issues macro, execution error.

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Currently, the initial socket connection timeout is depending on Linux kernel setting tcp_syn_retries. The timeout value is 2 ^ (tcp_sync_retries + 1) - 1 seconds. For the reasons below, we want to control the client-side socket timeout directly using configuration files. 

1. The default value of Linux  tcp_syn_retries is 6. It means the default timeout value is 127 seconds and too long in some scenarios. For example, when the user specifies a list of N bootstrap-servers and no connection has been built between the client and the servers, the least loaded node provider will poll all the server nodes specified by the user. If M servers in the bootstrap-servers list are offline, the client may take (127 * M) seconds to connect to the cluster. In the worst case when M = N - 1, the wait time can be several minutes.
2. Though we may set the default value of tcp_syn_retries smaller, we will then change the system level network behaviors, which might cause other issues.
3. Applications depending on KafkaAdminClient may want to robustly know and control the initial socket connect timeout, which can help throw corresponding exceptions in their layer.

Public Interfaces

We propose a new common client config

connections.setup.timeout.ms: The configuration controls the maximum amount of time the client will wait for the initial socket connection to be built. If the connection is not built before the timeout elapses the network client will close the socket channel. The default value will be 10 seconds.

Proposed Changes

I'm proposing to do a lazy socket connection time out. That is, we only check if we need to timeout a socket when we consider the corresponding node as a candidate in the node provider.

NetworkClient

public Node leastLoadedNode(long now)

This function iterates over all cached nodes and provides a node for the AdminClient to send the request. We can add our timeout checking logic inside the iteration, which won't downgrade the performance. 

ClusterConnectionStates

ClusterConnectionStates will keep the new config connection.setup.timeout.ms. We will also add a new public function

public boolean isConnectionSetupTimeout(String id, long now)

which will check if the connection setup is timeout or not based on connection.setup.timeout.ms and the current timestamp.

To Discuss

1. We can think about if we want to have different connect.timeout.ms for different clients.
2. What would be the default value for connect.timeout.ms (is 10s too short?)
3. Should the Selector / IdleExpiryManager throw an exception when the initial socket channel connection is timeout? Personally I don't think we need as we will continue trying the rest of the candidate nodes.

Compatibility, Deprecation, and Migration Plan

No impact

Rejected Alternatives