Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
KafkaStreams.java
	/* 
       Paused topologies will only skip over a) processing and b) punctuation.
	   Paused topologies will still poll Kafka consumers and commit offsets.

    */
    public void pause() 
	// Returns the state of the metadata.  The last run through the topology may still be executing.
    public boolean isPaused()
    public void resume()

...

The change is to add the above methods.  The implementation will manage internal metadata to achieve the desired results.

During task execution, only processing and punctuation will be skipped.  Kafka consumers will continue to poll and potentially fill up their buffers.  Commits will continue to happen on the schedule that they would have happened.

Changes to NamedTopologies / Modular Topologies.

Since NamedTopologies are internal, we do not have to provide details, but since a KIP for modular topologies is expected, let us discuss a few details.

First, there needs to be API calls for modular/named topologies to pause, resume, and check if a topology is paused.  Second, calling the above methods on an instance using modular topologies will pause/resume all of the topologies.  

Compatibility, Deprecation, and Migration Plan

...

Given that Kafka consumers are still reading data, it seems unnecessary to designate the change at the level of a state change

(Further, there is likely to be a future KIP for modular topologiesThe intersection of this KIP and that would be that one ought to be able to pause individual modular topologies.)