THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Sub-pages:
- Using Kafka Streams:
- Development:
- Internals:
Find more links about Kafka Streams at Kafka Ecosystem page.
Kafka Streams related KIPs:
Accepted
Below is a list of KIPs that are not release yet. Go to Kafka Streams KIP Overview for KIPs by release (including discarded KIPs).
Under discussion
- KIP-406: GlobalStreamThread should honor custom reset policy
- KIP-424: Allow suppression of intermediate events based on wall clock time
- KIP-439: Cleanup built-in Store interfaces
- KIP-448: Add State Stores Unit Test Support to Kafka Streams Test Utils
- KIP-508: Make Suppression State Queriable
- KIP-513: Distinguish between Key and Value serdes in scala wrapper library for kafka streams
- KIP-540: Implement per key stream time tracking
- KIP-560: Auto infer external topic partitions in stream reset tool
- KIP-598: Augment TopologyDescription with store and source / sink serde information
- KIP-634: Complementary support for headers and record metadata in Kafka Streams DSL
- KIP-647: Add ability to handle late messages in streams-aggregation
- KIP-655: Windowed Distinct Operation for Kafka Streams API
- KIP-669: Preserve Source Partition in Kafka Streams from context
- KIP-674: Metric Reporter to Aggregate Metrics in Kafka Streams
- KIP-675: Convert KTable to a KStream using the previous value
- KIP-677: Migrating Topology methods to the Builder pattern
- KIP-718: Make KTable Join on Foreign key unopinionated
- KIP-747 Add support for basic aggregation APIs
- KIP-807: Refactor KafkaStreams exposed metadata hierarchy
- KIP-816: Topology changes without local state reset
- KIP-819: Merge multiple KStreams in one operation
- KIP-839: Provide builders for KafkaProducer/KafkaConsumer and KafkaStreams
- KIP-857: Streaming recursion in Kafka Streams
- KIP-948: Allow custom prefix for internal topic names in Kafka Streams
- KIP-955: Add stream-table join on foreign key
- KIP-969: Support range Interactive Queries (IQv2) for Versioned State Stores
- KIP-989: RocksDB Iterator Metrics
- KIP-990: Capability to PAUSE Tasks on DeserializationException
- KIP-997: update WindowRangeQuery and unify WindowKeyQuery and WindowRangeQuery
Adopted
- inactive, not (fully) implemented (feel free to pick up)
- future release Kafka 2.1:
- KIP-150 - Kafka-Streams Cogroup
- KIP-206: Add support for UUID serialization and deserialization
- KIP-312: Add Overloaded StreamsBuilder Build Method to Accept java.util.Properties
KIP-321: Update TopologyDescription to better represent Source and Sink Nodes
KIP-331 Add default implementation to close() and configure() for Serializer, Deserializer and Serde
- Kafka 2.0:
- KIP-244: Add Record Header support to Kafka Streams Processor API
- KIP-245: Use Properties instead of StreamsConfig in KafkaStreams constructor
- KIP-251: Allow timestamp manipulation in Processor API
- KIP-261: Add Single Value Fetch in Window Stores
- KIP-265: Make Windowed Serde to public APIs
- KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils
- KIP-268: Simplify Kafka Streams Rebalance Metadata Upgrade
- KIP-270 - A Scala Wrapper Library for Kafka Streams
- KIP-274: Kafka Streams Skipped Records Metrics
- KIP-276 Add StreamsConfig prefix for different consumers
- KIP-284: Set default retention ms for Streams repartition topics to Long.MAX_VALUE
- KIP-292: Add transformValues() method to KTable
- KIP-295 Add Streams Configuration Allowing for Optional Topology Optimization
- KIP-303: Add Dynamic Routing in Streams Sink
- Kafka 1.1:
- KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner (partially implemented in v1.1)
- KIP-171 - Extend Consumer Group Reset Offset for Stream Application216: IQ should throw different exceptions for different errors
- KIP-205: Add all() and range() API to ReadOnlyWindowStore
- KIP-210 - Provide for custom error handling when Kafka Streams fails to produce
- KIP-220: Add AdminClient into Kafka Streams' ClientSupplier
- KIP-224: Add configuration parameter `retries` to Streams AP
- KIP-233: Simplify StreamsBuilder#addGlobalStore
- Kafka 1.0:
- KIP-120: Cleanup Kafka Streams builder API
- KIP 130: Expose states of active tasks to KafkaStreams public API
- KIP-138: Change punctuate semantics
- KIP 157 - Add consumer config options to streams reset tool
- KIP-160: Augment KStream.print(), KStream.writeAsText() to allow users pass in extra parameters in the printed string
- KIP-161: streams deserialization exception handlers
- KIP-167: Add interface for the state store restoration process
- KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs
- KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines
- KIP-198: Remove ZK dependency from Streams Reset Tool
- KIP-202 Move merge() from StreamsBuilder to KStream
- Kafka 0.11.0:
- KIP-114: KTable state stores and improved semantics
- KIP-123: Allow per stream/table timestamp extractor
- KIP-121: Add KStream peek method
- KIP-129: Streams Exactly-Once Semantics
- KIP-134: Delay initial consumer group rebalance
- KIP 155 - Add range scan for windowed state stores
- KIP 156 Add option "dry run" to Streams application reset tool
- Kafka 0.10.2:
- KIP-77: Improve Kafka Streams Join Semantics
- KIP-90: Remove zkClient dependency from Streams
- KIP-93: Improve invalid timestamp handling in Kafka Streams
- KIP-94: Session Windows
- KIP-97: Improved Kafka Client RPC Compatibility Policy
- KIP-99: Add Global Tables to Kafka Streams
- KIP-100: Relax Type constraints in Kafka Streams API
- KIP-104: Granular Sensors for Streams
Kafka 0.10.1:
Kafka 0.10.0:
...
- 258: Allow to Store Record Timestamps in RocksDB (partially implemented in v2.3)
- KIP-300: Add Windowed KTable API in StreamsBuilder (accepted but postponed because current code base does not allow to implement it as proposed)
- KIP-328: Ability to suppress updates for KTables (partially implemented in v2.1)
- KIP-557: Add emit on change support for Kafka Streams (
partially implemented in v2.6reverted again in 2.8.0, 2.7.1, and 2.6.2 due to potential data loss) - KIP-698: Add Explicit User Initialization of Broker-side State to Kafka Streams
- KIP-796: Interactive Query v2 (split up into multiple sub KIPs)
- KIP-844: Transactional State Stores (cf KIP-892: Transactional Semantics for StateStores)
- KIP-878: Internal Topic Autoscaling for Kafka Streams
- WIP (open PR / in review):
- KIP-759: Unneeded repartition canceling
- KIP-770: Replace "buffered.records.per.partition" & "cache.max.bytes.buffering" with "{statestore.cache}/{input.buffer}.max.bytes" (partially implemented in 3.4)
- KIP-813: Shareable State Stores
- KIP-892: Transactional Semantics for StateStores
- KIP-869: Improve Streams State Restoration Visibility (partially implemented in 3.5)
- next release Kafka 3.7 (merged):
- KIP-714: Client metrics and observability (includes some minor Kafka Streams changes)
- KIP-925: Rack aware task assignment in Kafka Streams (completed; partially implemented in 3.6)
- KIP-954: expand default DSL store configuration to custom types
- KIP-960: Support single-key_single-timestamp interactive queries for versioned state stores
- KIP-962: Relax non-null key requirement in Kafka Streams
- KIP-968: Support single-key_multi-timestamp Interactive Queries (IQv2) for Versioned State Stores
- KIP-985: Add reverseRange and reverseAll query over kv-store in IQv2
- KIP-988: Streams Standby Update Listener
- KIP-992: Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery
Inactive (feel free to pick up and resume the discussion):
- KIP-95: Incremental Batch Processing for Kafka Streams
- KIP-159: Introducing Rich functions to Streams
- KIP-165: Extend Interactive Queries for return latest update timestamp per key
- KIP-213 Support non-key joining in KTable
- KIP-216: IQ should throw different exceptions for different errors
- KIP-221: Repartition Topic Hints in Streams
- KIP-228 Negative record timestamp support
- KIP 230: Name Windowing Joins
- KIP-258: Allow to Store Record Timestamps in RocksDB
- KIP-262: Metadata should include number of state stores for task
- KIP-300: Add Windowed KTable API in StreamsBuilder
- KIP-307: Allow to define custom processor names with KStreams DSL
- 260: add primary join operation for Stream-Stream join (WIP) (draft)
- KIP-KIP-311: Async processing with dynamic scheduling in Kafka StreamsKIP-313: Add KStream.flatTransform and KStream.flatTransformValues
- KIP-314: KTable to GlobalKTable Bi-directional Join
KIP-319: Replace segments with segmentInterval in WindowBytesStoreSupplier
- KIP-324: Add method to get metrics() in AdminClient
- KIP-326: Schedulable KTable as Graph sourceKIP-328: Ability to suppress updates for KTables
- KIP-330: Add retentionPeriod in SessionBytesStoreSupplierKIP-335: Consider configurations for KafkaStreamsKIP-356: Add withCachingDisabled() to StoreBuilder
- KIP-358: Migrate Streams API to Duration instead of long ms timesKIP-362: Support dynamic gap session window
Inactive:
- KIP-95408: Incremental Batch Add Asynchronous Processing for To Kafka Streams
...
- KIP-457: Add DISCONNECTED status to Kafka Streams
- KIP-116: Add State Store Checkpoint Interval Configuration459: Improve KafkaStreams#close
- KIP-132: Augment KStream.print to allow extra parameters in the printed string (duplicated by KIP-160)
- KIP-147: Add missing type parameters to StateStoreSupplier factories and KGroupedStream/Table methods (covered by KIP-182)
...
- 463: Auto-configure non-default Serdes passed alongside the TopologyBuilder
- KIP-645: Replace Windows with a proper interface
- KIP-667: Remove deprecated methods from ReadOnlyWindowStore (no KIP needed)
- KIP-XXX: [Streams] Support multiple topic resolution in TopicNameExtractor (draft)