...

Sub-pages:

• Using Kafka Streams:
  
  • Usage Patterns
  • Application Reset Tool

  • Join Semantics

  • Streams FAQs
  • Data (Re)Processing Scenarios
• Development:
  • KIP overview by release
  • Development Plan
  • Discussions
  • Data (Re)Processing Scenarios
  • Kafka Streams DSL Grammar
  • Streams Runtime Architecture with New Threading
• Internals:
  
  • Data Management
  • Architecture
  • DSL Optimizer

Find more links about Kafka Streams at Kafka Ecosystem page.

Kafka Streams related KIPs:

Accepted

• future release Kafka 1.2:
  • KIP-150 - Kafka-Streams Cogroup
  • KIP-245: Use Properties instead of StreamsConfig in KafkaStreams constructor
  • KIP-251: Allow timestamp manipulation in Processor API
  • KIP-265: Make Windowed Serde to public APIs
  • KIP-268: Simplify Kafka Streams Rebalance Metadata Upgrade
• future release Kafka 1.1:
  
  • KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner (partially implemented in v1.1)
  • KIP-171 - Extend Consumer Group Reset Offset for Stream Application
  • KIP-205: Add all() and range() API to ReadOnlyWindowStore
  • KIP-210 - Provide for custom error handling when Kafka Streams fails to produce
  • KIP-220: Add AdminClient into Kafka Streams' ClientSupplier
  • KIP-224: Add configuration parameter `retries` to Streams AP
  • KIP-233: Simplify StreamsBuilder#addGlobalStore
  • KIP-239 Add queryableStoreName() to GlobalKTable
  • KIP-247: Add public test utils for Kafka Streams
• Kafka 1.0:
  • KIP-120: Cleanup Kafka Streams builder API
  • KIP 130: Expose states of active tasks to KafkaStreams public API
  • KIP-138: Change punctuate semantics
  • KIP 157 - Add consumer config options to streams reset tool
  • KIP-160: Augment KStream.print(), KStream.writeAsText() to allow users pass in extra parameters in the printed string
  • KIP-161: streams deserialization exception handlers
  • KIP-167: Add interface for the state store restoration process
  • KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs
  • KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines
  • KIP-198: Remove ZK dependency from Streams Reset Tool
  • KIP-202 Move merge() from StreamsBuilder to KStream
• Kafka 0.11.0:
  • KIP-114: KTable state stores and improved semantics
  • KIP-123: Allow per stream/table timestamp extractor
  • KIP-121: Add KStream peek method
  • KIP-129: Streams Exactly-Once Semantics
  • KIP-134: Delay initial consumer group rebalance
  • KIP 155 - Add range scan for windowed state stores
  • KIP 156 Add option "dry run" to Streams application reset tool
• Kafka 0.10.2:
  • KIP-77: Improve Kafka Streams Join Semantics
  • KIP-90: Remove zkClient dependency from Streams
  • KIP-93: Improve invalid timestamp handling in Kafka Streams
  • KIP-94: Session Windows
  • KIP-97: Improved Kafka Client RPC Compatibility Policy 
  • KIP-99: Add Global Tables to Kafka Streams
  • KIP-100: Relax Type constraints in Kafka Streams API
  • KIP-104: Granular Sensors for Streams

• Kafka 0.10.1:

  • KIP-62: Allow consumer to send heartbeats from a background thread
  • KIP-63: Unify store and downstream caching in streams
  • KIP-67: Queryable state for Kafka Streams
  • KIP-71: Enable log compaction and deletion to co-exist

• Kafka 0.10.0:

  • KIP-28: Add a processor client
  • KIP-32: Add timestamps to Kafka message

Under discussion

Below is a list of KIPs that are not release yet. Go to Kafka Streams KIP Overview for KIPs by release (including discarded KIPs).

Under discussion

• KIP-406: GlobalStreamThread should honor custom reset policy
• KIP-424: Allow suppression of intermediate events based on wall clock time
• KIP-439: Cleanup built-in Store interfaces
• KIP-448: Add State Stores Unit Test Support to Kafka Streams Test Utils
• KIP-508: Make Suppression State Queriable
• KIP-513: Distinguish between Key and Value serdes in scala wrapper library for kafka streams
• KIP-540: Implement per key stream time tracking
• KIP-560: Auto infer external topic partitions in stream reset tool
• KIP-598: Augment TopologyDescription with store and source / sink serde information
• KIP-634: Complementary support for headers and record metadata in Kafka Streams DSL
• KIP-647: Add ability to handle late messages in streams-aggregation
• KIP-655: Windowed Distinct Operation for Kafka Streams API
• KIP-669: Preserve Source Partition in Kafka Streams from context
• KIP-674: Metric Reporter to Aggregate Metrics in Kafka Streams
• KIP-675: Convert KTable to a KStream using the previous value
• KIP-677: Migrating Topology methods to the Builder pattern
• KIP-718: Make KTable Join on Foreign key unopinionated
• KIP-747 Add support for basic aggregation APIs
• KIP-807: Refactor KafkaStreams exposed metadata hierarchy
• KIP-816: Topology changes without local state reset
• KIP-819: Merge multiple KStreams in one operation
• KIP-839: Provide builders for KafkaProducer/KafkaConsumer and KafkaStreams
• KIP-857: Streaming recursion in Kafka Streams
• KIP-948: Allow custom prefix for internal topic names in Kafka Streams
• KIP-955: Add stream-table join on foreign key
• KIP-969: Support range Interactive Queries (IQv2) for Versioned State Stores 
• KIP-989: RocksDB Iterator Metrics
• KIP-990: Capability to PAUSE Tasks on DeserializationException
• KIP-997: update WindowRangeQuery and unify WindowKeyQuery and WindowRangeQuery

Adopted

• inactive, not (fully) implemented (feel free to pick up):
  • KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner (partially implemented in v1.1)
  • KIP-216: IQ should throw different exceptions for different errors
  • KIP-258: Allow to Store Record Timestamps in RocksDB (partially implemented in v2.3)
  • KIP-300: Add Windowed KTable API in StreamsBuilder (accepted but postponed because current code base does not allow to implement it as proposed) 
  • KIP-328: Ability to suppress updates for KTables (partially implemented in v2.1)
  • KIP-557: Add emit on change support for Kafka Streams (partially implemented in v2.6 reverted again in 2.8.0, 2.7.1, and 2.6.2 due to potential data loss)
  • KIP-698: Add Explicit User Initialization of Broker-side State to Kafka Streams
  • KIP-796: Interactive Query v2 (split up into multiple sub KIPs)
  • KIP-844: Transactional State Stores (cf KIP-892: Transactional Semantics for StateStores)
  • KIP-878: Internal Topic Autoscaling for Kafka Streams
• WIP (open PR / in review):
  
  • KIP-759: Unneeded repartition canceling
  • KIP-770: Replace "buffered.records.per.partition" & "cache.max.bytes.buffering" with "{statestore.cache}/{input.buffer}.max.bytes" (partially implemented in 3.4)
  • KIP-813: Shareable State Stores
  • KIP-892: Transactional Semantics for StateStores
  • KIP-869: Improve Streams State Restoration Visibility (partially implemented in 3.5)
• next release Kafka 3.7 (merged):
  • KIP-714: Client metrics and observability (includes some minor Kafka Streams changes)
  • KIP-925: Rack aware task assignment in Kafka Streams (completed; partially implemented in 3.6)
  • KIP-954: expand default DSL store configuration to custom types
  • KIP-960: Support single-key_single-timestamp interactive queries for versioned state stores
  • KIP-962: Relax non-null key requirement in Kafka Streams
  • KIP-968: Support single-key_multi-timestamp Interactive Queries (IQv2) for Versioned State Stores 
  • KIP-985: Add reverseRange and reverseAll query over kv-store in IQv2
  • KIP-988: Streams Standby Update Listener
  • KIP-992: Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery

Inactive (feel free to pick up and resume the discussion):

• KIP-95: Incremental Batch Processing for Kafka Streams
• KIP-159: Introducing Rich functions to Streams
• KIP-165: Extend Interactive Queries for return latest update timestamp per key
• KIP-213 Support non-key joining in KTable
• KIP-216: IQ should throw different exceptions for different errors
• KIP-221: Repartition Topic Hints in Streams
• KIP-228 Negative record timestamp support
• KIP 230: Name Windowing Joins
• KIP-244: Add Record Header support to Kafka Streams
• KIP-258: Allow to Store Record Timestamps in RocksDB
• KIP-261: Add Single Value Fetch in Window Stores
• KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils
• KIP-270 - A Scala Wrapper Library for Kafka Streams

Inactive:

• KIP-95: Incremental Batch Processing for Kafka Streams

Discarded:

• KIP-116: Add State Store Checkpoint Interval Configuration
• KIP-132: Augment KStream.print to allow extra parameters in the printed string (duplicated by KIP-160)
• KIP-147: Add missing type parameters to StateStoreSupplier factories and KGroupedStream/Table methods (covered by KIP-182)

...

• 260: add primary join operation for Stream-Stream join (WIP) (draft)
• KIP-311: Async processing with dynamic scheduling in Kafka Streams
• KIP-314: KTable to GlobalKTable Bi-directional Join
• KIP-326: Schedulable KTable as Graph source
• KIP-335: Consider configurations for KafkaStreams
• KIP-362: Support dynamic gap session window
• KIP-408: Add Asynchronous Processing To Kafka Streams
• KIP-457: Add DISCONNECTED status to Kafka Streams
• KIP-459: Improve KafkaStreams#close
• KIP-463: Auto-configure non-default Serdes passed alongside the TopologyBuilder
• KIP-645: Replace Windows with a proper interface
• KIP-667: Remove deprecated methods from ReadOnlyWindowStore (no KIP needed)
• KIP-XXX: [Streams] Support multiple topic resolution in TopicNameExtractor (draft)