Status
Current state: Under Discussion
Discussion thread: here
JIRA: KAFKA-13873
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
In order to reduce resources used or modify data pipelines, users may want to pause processing temporarily. Presently, this would require stopping the entire KafkaStreams instance (or instances).
This KIP proposes the addition of the ability to pause and resume topologies. When the need to pause processing has passed, then users should be able to resume processing.
Public Interfaces
This proposal adds public methods to pause and resume KafkaStreams clients.
KafkaStreams.java /* Paused topologies will still poll Kafka consumers and commit offsets. */ public void pause() public boolean isPaused() public void resume()
Proposed Changes
The change is to add the above methods. The implementation will manage internal metadata to achieve the desired results.
Compatibility, Deprecation, and Migration Plan
Since this KIP is additive, there are no compatibility concerns.
Rejected Alternatives
Alternative: Adding a new PAUSED state.
Given that Kafka consumers are still reading data, it seems unnecessary to designate the change at the level of a state change.