You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 13 Next »

Status

Current state: Under discussion.

Discussion thread: here  (Not happening yet)

JIRA: here

Motivation

Transient network issues such as slow DNS resolution might lead to client instantiation failing.  The behavior is disruptive because the application owner must either implement retry logic or manually restart the application.  After all, it is not very straightforward to address network issue because of the following reasons.

  1. Failed DNS resolution throws a ConfigException, which is not descriptive of the actual problem.  While the error message is fine, but catch and retrying based on the error message is not ideal.

  2. One of the common problems we encountered was DNS resolution.  In a dynamic cloud environment, it can take tens of minutes before the bootstrap server is registered to the DNS server and it is convenient to provide a grace period before failing the client.

Public Interfaces

See Proposed Changes.

  1. New config to timeout the bootstrap connection.
  2. A new exception type of prescribing for this specific issue.
  3. Additional logging upon failure to bootstrap.
  4. A change in failure condition.

Proposed Changes

Public Facing Changes

  • Config: bootstrap.connection.timeout.ms
  • Exception: BootstrapConnectionException extends KafkaException (non-retriable)
  • Logging: log.WARN("Unable to bootstrap, retry in {} ms.")

Internal Changes

  • Client Constructor: Only parse the bootstrap config and validate its format there

  • NetworkClient:

    • Bootstrapping should now occur in the poll method before attempting to update the metadata. This includes resolving the addresses and bootstrapping the metadata.

    • Logs an error message with failed bootstrap process

    • If the timeout exceeds, throw a BootstrapConnectionException, which is non-retriable
  • Consumer Client: Bootstrap logic will be moved.
  • Producer Client: Bootstrap logic will be moved.
  • Admin Client: This will be moved.

New Configuration

bootstrap.connection.timeout.ms

The amount of time clients can try to establish a connection to the bootstrap server and resolve for the IP address. If the time exceeds this value, a BootstrapConnectionException will be thrown.

Note: the default value is up for discussion. It can be 0, which is the same as the current behavior.  Exit upon the first retry.

Type:long
Default:300000 (5 minutes)
Valid Values:0 - LONG_MAX
Importance:high

New Error

Name: BootstrapConnectionException extends KafkaException

Message: "Unable to establish a connection to the bootstrap server in {}ms."

Type: Non-retriable.

Compatibility, Deprecation, and Migration Plan

  • Client Behaviors

    • Clients won’t attempt to resolve the bootstrap addresses upon initialization.

    • Clients won’t exit fatally if DNS resolution fails.

    • KafkaConsumer: Users must poll to retry the lookup if it fails.

    • KafkaAdminClient: Users will need to resend the request if failing.

    • KafkaProducer: The sender loop should already be polling continuously.

    Exception Handling

    • Failed DNS resolution will result in NetworkException

Case Study

To help illustrate the proposed changes, we provide some examples of how clients might behave in different scenarios.

KafkaConsumer

Case 1: Unable to connect to the bootstrap (For example: misconfiguration)

Suppose the user instantiates a KafkaConsumer with an invalid bootstrap config. When the user invokes assign() and starts poll(), the poll() method will continue to return empty ConsumerRecords and log a warning message.

The user can continue to retry for the configured duration. After the bootstrap timeout expires, the client will throw a BootstrapConnectionException.

Case 2: Transient Network Issue (For example: transient DNS failure)

Now, suppose the user instantiates a KafkaConsumer with a valid bootstrap config, but there is a transient network issue, such as slow DNS resolution.

When the user starts poll(), the poll() method will return an empty ConsumerRecord and log a warning message.

The user can continue to retry, and the network issue will be successfully resolved after some time. The KafkaConsumer will then continue to function normally.

KafkaProducer

Case 1: Unable to connect to the bootstrap (For example: misconfiguration)

Suppose the user instantiates a KafkaProducer with an invalid bootstrap config. As the produce is instantiated, the sender thread starts running.  A warning message is logged everytime the NetworkClient tries to poll().

If the user tries to produce messages, the producer callback may be completed with a TimeoutException until the bootstrap timeout runs out.

Eventually, a BootstrapConnectionException will be thrown.

Case 2: Transient Network Issue (For example: transient DNS failure)

Now, suppose the user instantiates a KafkaProducer with a valid bootstrap config, but there is a transient network issue. As the sender thread starts running, a warning message is logged upon trying to bootstrap the client.

If the network issue is resolved before the user tries to produce a message, only warning messages will be logged.

If the user tries to produce a message before the issue is resolved, the sender callback will be completed with a TimeoutException if the network issue persists. The send will be completed normally if the network issue is resolved before exhausting the max.block.ms.

AdminClient

Case 1: Unable to connect to the bootstrap (For example: misconfiguration)

  1. The user instantiates a new admin client.
  2. If the user makes admin client API calls:
    1. If the API timeout before the bootstrap timeout expires, a TimeoutException will be thrown upon invoking .get()
    2. If the bootstrap timeout expires, a BootstrapConnectionException will be thrown.
    3. Any API calls will not be completed.

Case 2: Transient Network Issue (For example: transient DNS failure)

  1. The user instantiates a new admin client.
  2. If the API timeout before the connection is resolved, a TimeoutException will be thrown upon invoking .get() for the API call results
  3. Eventually, the API calls should go through.

Test Plan

  1. NetworkClient

    1. Test DNS resolution upon its initial poll

    2. Test if the right exception type is thrown

  2. Existing clients (Consumer, Producer, AdminClient)

    1. Test successful bootstrapping upon retrying

Rejected Alternatives

  1. Allow the application owner to specify a retry period. The clients will fail after exceeding the timeout. The default set to 0s, which makes retry an opt-in config.

    1. Pros: Allows users to have more control over how long to retry

    2. Cons: Require a new config; client instantiation can block.

  2. No retry. Let the application owner handle the DNS resolution exception. This means we would still throw a DNSLookupException upon failing.

    1. Pros: No additional config is needed

    2. Cons: This is a behavioral change, and the application owner might need to rewrite the exception handling, i.e. catching the DNS failure logic.

  3. Not throwing an exception but letting NetworkClient retry on poll.
    1. Pros: No compatibility break. No additional exception handling logic, the network client will just log the error and continue to retry upon the next poll
    2. Cons: The discussion thread mentioned that it wouldn't fail upon starting.
  • No labels