Status
Current state: Under Discussion
Discussion thread: here
JIRA: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Currently, when GlobalStreamThread dies from a fatal error, an InvalidOffsetException will be forwarded to the user which is used to indicate that something wrong had occurred. We have improved upon the durability of GlobalStreamThread previously by wiping all data that had already been processed and rebuilding the store entirely (which was done in KAFKA-6121). However, we should also give the user the option of choosing their reset policy which up to this time, they had no chance of selecting.
Public Interfaces
A new reset policy(s) will be added which will allow the user some flexibility in determining the resilience of GlobalStreamThread. It should also be noted that Topology.addGlobalStore
does not allow the user to decide which policy to use. This method will potentially have to be extended.
public class Topology { // newly added public synchronized <K, V> Topology addGlobalStore(final AutoOffsetReset offsetReset, final StoreBuilder<?> storeBuilder, final String sourceName, final TimestampExtractor timestampExtractor, final Deserializer<K> keyDeserializer, final Deserializer<V> valueDeserializer, final String topic, final String processorName, final ProcessorSupplier<K, V> stateUpdateSupplier) { } }
Proposed Changes
- Use a new config- global.auto.offset.reset to set reset policy for global stream thread or use the existing one- auto.offset.reset to reset in both StreamThread and GlobalStreamThread.
Compatibility, Deprecation, and Migration Plan
- The current
Topology.addGlobalStore
method will be extended. We will add a second version ofaddGlobalStore
with extra input parameters while the original method calls the added method with default values. This would not force the user to change their preexisting code in any way.
Rejected Alternatives
N/A