...

Kafka Streams related KIPs:

Below is a list of KIPs that are not release yet. Go to Kafka Streams KIP Overview for KIPs by release (including discarded KIPs).

Under discussion

• KIP-216: IQ should throw different exceptions for different errors
• KIP-228 Negative record timestamp support
• KIP-399: Extend ProductionExceptionHandler to cover serialization exceptions
• KIP-424: Allow suppression of intermediate events based on wall clock time
• KIP-439: Cleanup built-in Store interfaces
• KIP-447: Producer scalability for exactly once semantics
• KIP-448: Add State Stores Unit Test Support to Kafka Streams Test Utils
• KIP-513: Distinguish between Key and Value serdes in scala wrapper library for kafka streams
• KIP-540: Implement per key stream time tracking
• KIP-557: Add emit on change support for Kafka Streams

Adopted

• inactive (feel free to pick up):
  • KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner (partially implemented in v1.1)
  • KIP-258: Allow to Store Record Timestamps in RocksDB (partially implemented in v2.3)
  • KIP-300: Add Windowed KTable API in StreamsBuilder (accepted but postponed because current code base does not allow to implement it as proposed) 
  • KIP-328: Ability to suppress updates for KTables (partially implemented in v2.1)
  • KIP-446: Add changelog topic configuration to KTable suppress
  • KIP-471: Expose RocksDB Metrics in Kafka Streams (partially implemented in v2.4; completion deferred to next major release)
  • KIP-478 - Strongly typed Processor API
• future release Kafka 2.5:
  • KIP-150 - Kafka-Streams Cogroup
  • KIP-221: Enhance DSL with Connecting Topic Creation and Repartition Hint
  • KIP-401: TransformerSupplier/ProcessorSupplier StateStore connecting
  • KIP-441: Smooth Scaling Out for Kafka Streams (WIP)
  • KIP-444: Augment metrics for Kafka Streams (partially implemented in v2.4)
  • KIP-466: Add support for List<T> serialization and deserialization
  • KIP-523: Add KStream#toTable to the Streams DSL
  • KIP-527: Add VoidSerde to Serdes
  • KIP-530: Consider renaming 'UsePreviousTimeOnInvalidTimeStamp' class to 'UsePartitionTimeOnInvalidTimeStamp'
  • KIP-535: Allow state stores to serve stale reads during rebalance
• Kafka 2.4:
  • KIP-213 Support non-key joining in KTable
  • KIP-307: Allow to define custom processor names with KStreams DSL (completed; partially implemented in v2.3)
  • KIP-345: Introduce static membership protocol to reduce consumer rebalances (completed; partially implemented in v2.3)
  • KIP-429: Kafka Consumer Incremental Rebalance Protocol
  • KIP-444: Augment metrics for Kafka Streams (partially implemented; completion WIP for 2.5)
  • KIP-470: TopologyTestDriver test input and output usability improvements
  • KIP-471: Expose RocksDB Metrics in Kafka Streams (partially implemented; completion blocked)
  • KIP-474: To deprecate WindowStore#put(key, value)
  • KIP-479: Add StreamJoined config object to Join
  • KIP-528: Deprecate PartitionGrouper configuration and interface
• Kafka 2.3: 
  • KIP-258: Allow to Store Record Timestamps in RocksDB (partially implemented; not completed yet → inactive)
  • KIP-307: Allow to define custom processor names with KStreams DSL (partially implemented; completed in 2.4)
  • KIP-313: Add KStream.flatTransform and KStream.flatTransformValues (completed; partially implemented in v2.2)
  • KIP-331 Add default implementation to close() and configure() for Serializer, Deserializer and Serde
  • KIP-345: Introduce static membership protocol to reduce consumer rebalances (partially implemented; completed in 2.4)
  • KIP-428: Add in-memory window store
  • KIP-442: Return to default max poll interval in Streams
  • KIP-443: Return to default segment.ms and segment.index.bytes in Streams repartition topics
  • KIP-445: In-memory Session Store
  • KIP-453: Add close() method to RocksDBConfigSetter
  • KIP-462: Use local thread id for KStreams
• Kafka 2.2:
  

  • KIP-313: Add KStream.flatTransform and KStream.flatTransformValues (partially implemented; completed in 2.3)

  • KIP-376: Implement AutoClosable on appropriate classes that want to be used in a try-with-resource statement

  • KIP-386: Standardize on Min/Avg/Max metrics' default value

  • KIP-393: Time windowed serde to properly deserialize changelog input topic

  • KIP-414: Expose Embedded ClientIds in Kafka Streams

  • KIP-420: Add Single Value Fetch in Session Stores

• Kafka 2.1:
  
  • KIP-206: Add support for UUID serialization and deserialization
  • KIP-312: Add Overloaded StreamsBuilder Build Method to Accept java.util.Properties

  • KIP-319: Replace segments with segmentInterval in WindowBytesStoreSupplier

  • KIP-321: Update TopologyDescription to better represent Source and Sink Nodes

  • KIP-324: Add method to get metrics() in AdminClient

  • KIP-328: Ability to suppress updates for KTables (partially implemented; not completed yet → inactive)

  • KIP-330: Add retentionPeriod in SessionBytesStoreSupplier

  • KIP-353: Improve Kafka Streams Timestamp Synchronization

  • KIP-356: Add withCachingDisabled() to StoreBuilder

  • KIP-358: Migrate Streams API to Duration instead of long ms times

  • KIP-365: Materialized, Serialized, Joined, Consumed and Produced with implicit Serde

  • KIP-366: Make FunctionConversions deprecated

  • KIP-372: Naming Repartition Topics for Joins and Grouping

• Kafka 2.0:
  • KIP-244: Add Record Header support to Kafka Streams Processor API
  • KIP-245: Use Properties instead of StreamsConfig in KafkaStreams constructor
  • KIP-251: Allow timestamp manipulation in Processor API
  • KIP-261: Add Single Value Fetch in Window Stores
  • KIP-265: Make Windowed Serde to public APIs
  • KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils
  • KIP-268: Simplify Kafka Streams Rebalance Metadata Upgrade
  • KIP-270 - A Scala Wrapper Library for Kafka Streams
  • KIP-274: Kafka Streams Skipped Records Metrics
  • KIP-276 Add StreamsConfig prefix for different consumers
  • KIP-284: Set default retention ms for Streams repartition topics to Long.MAX_VALUE
  • KIP-292: Add transformValues() method to KTable
  • KIP-295 Add Streams Configuration Allowing for Optional Topology Optimization
  • KIP-303: Add Dynamic Routing in Streams Sink
• Kafka 1.1:
  
  • KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner (partially implemented; not completed yet → inactive)
  • KIP-171 - Extend Consumer Group Reset Offset for Stream Application
  • KIP-205: Add all() and range() API to ReadOnlyWindowStore
  • KIP-210 - Provide for custom error handling when Kafka Streams fails to produce
  • KIP-220: Add AdminClient into Kafka Streams' ClientSupplier
  • KIP-224: Add configuration parameter `retries` to Streams AP
  • KIP-233: Simplify StreamsBuilder#addGlobalStore
  • KIP-239 Add queryableStoreName() to GlobalKTable
  • KIP-247: Add public test utils for Kafka Streams
• Kafka 1.0:
  • KIP-120: Cleanup Kafka Streams builder API
  • KIP 130: Expose states of active tasks to KafkaStreams public API
  • KIP-138: Change punctuate semantics
  • KIP 157 - Add consumer config options to streams reset tool
  • KIP-160: Augment KStream.print(), KStream.writeAsText() to allow users pass in extra parameters in the printed string
  • KIP-161: streams deserialization exception handlers
  • KIP-167: Add interface for the state store restoration process
  • KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs
  • KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines
  • KIP-198: Remove ZK dependency from Streams Reset Tool
  • KIP-202 Move merge() from StreamsBuilder to KStream
• Kafka 0.11.0:
• KIP-216: IQ should throw different exceptions for different errors
• KIP-228 Negative record timestamp support
• KIP-424: Allow suppression of intermediate events based on wall clock time
• KIP-439: Cleanup built-in Store interfaces
• KIP-447: Producer scalability for exactly once semantics
  • KIP-114: KTable state stores and improved semantics
  • KIP-123: Allow per stream/table timestamp extractor
  • KIP-121: Add KStream peek method
  • KIP-129: Streams Exactly-Once Semantics
  • KIP-134: Delay initial consumer group rebalance
  • KIP 155 - Add range scan for windowed state stores
  • KIP 156 Add option "dry run" to Streams application reset tool
• Kafka 0.10.2:
  • KIP-77: Improve Kafka Streams Join Semantics
  • KIP-90: Remove zkClient dependency from Streams
  • KIP-93: Improve invalid timestamp handling in Kafka Streams
  • KIP-94: Session Windows
  • KIP-97: Improved Kafka Client RPC Compatibility Policy 
  • KIP-99: Add Global Tables to Kafka Streams
  • KIP-100: Relax Type constraints in Kafka Streams API
  • KIP-104: Granular Sensors for Streams

• Kafka 0.10.1:

  • KIP-62: Allow consumer to send heartbeats from a background thread
  • KIP-63: Unify store and downstream caching in streams
  • KIP-67: Queryable state for Kafka Streams
  • KIP-71: Enable log compaction and deletion to co-exist

• Kafka 0.10.0:

  • KIP-28: Add a processor client
  • KIP-32: Add timestamps to Kafka message

Under discussion

• KIP-448: Add State Stores Unit Test Support to Kafka Streams Test Utils
• KIP-513: Distinguish between Key and Value serdes in scala wrapper library for kafka streams
• KIP-540: Implement per key stream time tracking

Inactive (feel free to pick up):

• KIP-95: Incremental Batch Processing for Kafka Streams
• KIP-159: Introducing Rich functions to Streams
• KIP-260: add primary join operation for Stream-Stream join (WIP) (draft)
• KIP-311: Async processing with dynamic scheduling in Kafka Streams
• KIP-314: KTable to GlobalKTable Bi-directional Join
• KIP-326: Schedulable KTable as Graph source
• KIP-335: Consider configurations for KafkaStreams
• KIP-362: Support dynamic gap session window
• KIP-378: Enable Dependency Injection for Kafka Streams handlers
• KIP-399: Extend ProductionExceptionHandler to cover serialization exceptionsKIP-406: GlobalStreamThread should honor custom reset policy
• KIP-408: Add Asynchronous Processing To Kafka Streams
• KIP-418: A method-chaining way to branch KStream
• KIP-450: Sliding Window Aggregations in the DSL
• KIP-457: Add DISCONNECTED status to Kafka Streams
• KIP-459: Improve KafkaStreams#close
• KIP-463: Auto-configure non-default Serdes passed alongside the TopologyBuilder
• KIP-508: Make Suppression State Queriable
• KIP-XXX: [Streams] Support multiple topic resolution in TopicNameExtractor (draft)

Discarded:

• KIP-116: Add State Store Checkpoint Interval Configuration
• KIP-132: Augment KStream.print to allow extra parameters in the printed string (duplicated by KIP-160)
• KIP-147: Add missing type parameters to StateStoreSupplier factories and KGroupedStream/Table methods (covered via KIP-182)
• KIP-165: Extend Interactive Queries for return latest update timestamp per key (covered via KIP-258)
• KIP 230: Name Windowing Joins (covered via KIP-372: Naming Repartition Topics for Joins and Grouping)
• KIP-259: Improve Streams DSL Timestamp Propagation Semantics (change did not require a KIP)
• KIP-262: Metadata should include number of state stores for task
• KIP-364: Remove implicit Materialized, Consumed and Produced (covered via KIP-365: Materialized, Serialized, Joined, Consumed and Produced with implicit Serde)
  
• KIP-451: Make TopologyTestDriver output iterable (subsumed by KIP-456)
• KIP-456: Helper classes to make it simpler to write test logic with TopologyTestDriver (subsumed by KIP-470)
• KIP-472: Add header to RecordContext/KafkaProducer
• KIP-485: Make topic optional when using through() operations in DSL (subsumed by KIP-221)