Status
Current state: "Accepted" [VOTE] KIP-224: Add configuration parameters `retries` and `retry.backoff.ms` to Streams API
Discussion thread: [DISCUSS] KIP-224: Add configuration parameters `retries` and `retry.backoff.ms` to Streams API
JIRA: KAFKA-6122
Released: 1.1.0
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Streams API can fail with a TimeoutException if a global state (or global KTable) is used. This is an issue if the brokers are temporarily not available, as no retry strategy is provided, and thus the GlobalStreamThread
(and consequently the whole KafkaStreams instance) dies. To make Streams API robust against such issues, it would be required to implement a configurable retry strategy.
Public Interfaces
We suggest to add a new configuration parameters to StreamsConfig
namely "retries" similar to KafkaConsumer and KafkaProducer. To keep the current "fail-fast" strategy for default config, default value for "retries" could be 0. Furthermore, we would use already existing parameter "retry.backoff.ms" for this retry strategy.
Proposed Changes
We would apply both parameters in GlobalStateManagerImpl to guard against TimeoutException for KafkaCosumer#partitionsFor and KafkaConsumer#endOffsets.
To align with current design, we would also pass the new parameter to producer and consumer if there is no "producer." or "consumer." prefix. Thus, users can still configure different values for consumer/producer if needed.
Compatibility, Deprecation, and Migration Plan
- This change is backwards compatible as we only add new parameters and also keep the current behavior with default parameter settings.
Test Plan
We add tests with mocking the global consumer client to throw TimeoutException to check if retry parameters are respected.
Rejected Alternatives
None.