Kafka Improvement Proposals

This page describes a proposed Kafka Improvement Proposal (KIP) process for proposing a major change to Kafka.

To create your own KIP, click on Create KIP. If you don't have permission, please send an email with your Wiki ID to dev@kafka.apache.org and request permission (http://kafka.apache.org/contact). Also add an entry to the table Kafka Improvement Proposals#KIPs under discussion (for Streams API KIPs, please also add it to Kafka Streams sub page).

Purpose

We want to make Kafka a core architectural component for users. We also support a large number of integrations with other tools, systems, and clients. Keeping this kind of usage health requires a high level of compatibility between releases — core architectural elements can't break compatibility or shift functionality from release to release. As a result each new major feature or public api has to be done in a way that we can stick with it going forward.

This means when making this kind of change we need to think through what we are doing as best we can prior to release. And as we go forward we need to stick to our decisions as much as possible. All technical decisions have pros and cons so it is important we capture the thought process that lead to a decision or design to avoid flip-flopping needlessly.

Hopefully we can make these proportional in effort to their magnitude — small changes should just need a couple brief paragraphs, whereas large changes need detailed design discussions.

This process also isn't meant to discourage incompatible changes — proposing an incompatible change is totally legitimate. Sometimes we will have made a mistake and the best path forward is a clean break that cleans things up and gives us a good foundation going forward. Rather this is intended to avoid accidentally introducing half thought-out interfaces and protocols that cause needless heartburn when changed. Likewise the definition of "compatible" is itself squishy: small details like which errors are thrown when are clearly part of the contract but may need to change in some circumstances, likewise performance isn't part of the public contract but dramatic changes may break use cases. So we just need to use good judgement about how big the impact of an incompatibility will be and how big the payoff is.

What is considered a "major change" that needs a KIP?

Any of the following should be considered a major change:

Any major new feature, subsystem, or piece of functionality
Any change that impacts the public interfaces of the project

What are the "public interfaces" of the project?

All of the following are public interfaces that people build around:

Binary log format
The network protocol and api behavior
Any class for which the build generates Javadoc. This includes (but is not limited to; look for javadoc {} sections in build.gradle for the complete list):
- org/apache/kafka/common/serialization
- org/apache/kafka/common
- org/apache/kafka/common/errors
- org/apache/kafka/clients/producer
- org/apache/kafka/clients/consumer (eventually, once stable)
Configuration, especially client configuration
Monitoring
Command line tools and arguments

Not all compatibility commitments are the same. We need to spend significantly more time on log format and protocol as these break code in lots of clients, cause downtime releases, etc. Public apis are next as they cause people to rebuild code and lead to compatibility issues in large multi-dependency projects (which end up requiring multiple incompatible versions). Configuration, monitoring, and command line tools can be faster and looser — changes here will break monitoring dashboards and require a bit of care during upgrades but aren't a huge burden.

For the most part monitoring, command line tool changes, and configs are added with new features so these can be done with a single KIP.

What should be included in a KIP?

A KIP should contain the following sections:

Motivation: describe the problem to be solved
Proposed Change: describe the new thing you want to do. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences, depending on the scope of the change.
New or Changed Public Interfaces: impact to any of the "compatibility commitments" described above. We want to call these out in particular so everyone thinks about them.
Migration Plan and Compatibility: if this feature requires additional support for a no-downtime upgrade describe how that will work
Rejected Alternatives: What are the other alternatives you considered and why are they worse? The goal of this section is to help people understand why this is the best solution now, and also to prevent churn in the future when old alternatives are reconsidered.

Who should initiate the KIP?

Anyone can initiate a KIP but you shouldn't do it unless you have an intention of getting the work done to implement it (otherwise it is silly).

Process

Here is the process for making a KIP:

Click Create KIP. Take the next available KIP number and give your proposal a descriptive heading. e.g. "KIP 42: Allow Infinite Retention With Bounded Disk Usage".
Fill in the sections as described above
Start a [DISCUSS] thread on the Apache mailing list. Please ensure that the subject of the thread is of the format [DISCUSS] KIP-{your KIP number} {your KIP heading} and the body contains a link to your new KIP. The discussion should happen on the mailing list, not on the wiki, since the wiki comment system doesn't work well for larger discussions. In the process of the discussion you may update the proposal. You should let people know the changes you are making. When you feel you have a finalized proposal
Once the proposal is finalized call a [VOTE] to have the proposal adopted. These proposals are more serious than code changes and more serious even than release votes. The criteria for acceptance is lazy majority. The vote should remain open for at least 72 hours.
Please update the KIP wiki page, and the index below, to reflect the current stage of the KIP after a vote. This acts as the permanent record indicating the result of the KIP (e.g., Accepted or Rejected). Also report the result of the KIP vote to the voting thread on the mailing list so the conclusion is clear.

KIP round-up

Next KIP Number: 581

Use this number as the identifier for your KIP and increment this value.

Adopted KIPs

Please insert new rows in sorted order (descending by KIP number).

	KIP (please keep this sorted by KIP number)	Release
1	KIP-579: new exception on min.insync.replicas > replication.factor	2.6.0 (WIP)
2	KIP-557: Add emit on change support for Kafka Streams	2.6.0 (WIP)
3	KIP-570: Add leader epoch in StopReplicaRequest	2.6.0 (WIP)
4	KIP-568: Explicit rebalance triggering on the Consumer	2.6.0 (WIP)
5	KIP-562: Allow fetching a key from a single partition rather than iterating over all the stores on an instance	2.5.0 (WIP)
6	KIP-559: Make the Kafka Protocol Friendlier with L7 Proxies	2.5.0
7	KIP-558: Track the set of actively used topics by connectors in Kafka Connect	2.5.0
8	KIP-555: Deprecate direct Zookeeper access in Kafka administrative tools	2.5.0
9	KIP-553: Disable all SSL protocols except TLSV1.2 by default.	2.5.0
10	KIP-551: Expose disk read and write metrics	2.5.0 (WIP)
11	KIP-546: Add quota-specific APIs to the Admin Client, redux	2.6.0 (WIP)
12	KIP-544: Make metrics exposed via JMX configurable	2.5.0 (WIP)
13	KIP-543: Expand ConfigCommand's non-ZK functionality	2.5.0
14	KIP-541: Create a fetch.max.bytes configuration for the broker	2.5.0
15	KIP-538: Add a metric tracking the number of open connections with a given SSL cipher type	2.5.0
16	KIP-537: Increase default zookeeper session timeout	2.5.0
17	KIP-535: Allow state stores to serve stale reads during rebalance	2.5.0
18	KIP-534: Reorganize checkpoint system in log cleaner to per partition	WIP
19	KIP-533: Add default api timeout to AdminClient	2.5.0
20	KIP-532: Broker Consumer Lag metrics in size and time	WIP
21	KIP-531: Drop support for Scala 2.11 in Kafka 2.5	2.5.0 (WIP)
22	KIP-530: Consider renaming 'UsePreviousTimeOnInvalidTimeStamp' class to 'UsePartitionTimeOnInvalidTimeStamp'	2.5.0 (WIP)
23	KIP-528: Deprecate PartitionGrouper configuration and interface	2.4.0
24	KIP-527: Add VoidSerde to Serdes	2.5.0
25	KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics	2.5.0
26	KIP-525 - Return topic metadata and configs in CreateTopics response	2.4.0
27	KIP-524: Allow users to choose config source when describing configs	2.5.0 (WIP)
28	KIP-523: Add KStream#toTable to the Streams DSL	2.5.0 (WIP)
29	KIP-521: Enable redirection of Connect's log4j messages to a file by default	2.4.0
30	KIP-518: Allow listing consumer groups per state	2.6.0 (WIP)
31	KIP-517: Add consumer metrics to observe user poll behavior	2.4.0
32	KIP-515: Enable ZK client to use the new TLS supported authentication	2.5.0
33	KIP-514:Add a bounded flush() API to Kafka Producer	2.5.0
34	KIP-511: Collect and Expose Client's Name and Version in the Brokers	2.4.0 (protocol) / 2.5.0 (metric)
35	KIP-507: Securing Internal Connect REST Endpoints	2.4.0
36	KIP-504 - Add new Java Authorizer Interface	2.4.0
37	KIP-503: Add metric for number of topics marked for deletion	2.4.0
38	KIP-501: Avoid out-of-sync or offline partitions when follower fetch requests not processed in time	WIP
39	KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
40	KIP-497: Add inter-broker API to alter ISR	2.6.0 (WIP)
41	KIP-496: Administrative API to delete consumer offsets	2.4.0
42	KIP-495: Dynamically Adjust Log Levels in Connect	2.4.0
43	KIP-492: Add java security providers in Kafka Security config	2.4.0
44	KIP-488: Clean up Sum,Count,Total Metrics	2.4.0
45	KIP-484: Expose metrics for group and transaction metadata loading duration	2.4.0
46	KIP-482: The Kafka Protocol should Support Optional Tagged Fields	2.4.0
47	KIP-481: SerDe Improvements for Connect Decimal type in JSON	2.4.0
48	KIP-480: Sticky Partitioner	2.4.0
49	KIP-479: Add StreamJoined config object to Join	2.4.0
50	KIP-478 - Strongly typed Processor API	WIP
51	KIP-476: Add Java AdminClient Interface	2.4.0
52	KIP-475: New Metrics to Measure Number of Tasks on a Connector	2.4.0
53	KIP-474: To deprecate WindowStore#put(key, value)	2.4.0
54	KIP-471: Expose RocksDB Metrics in Kafka Streams	2.4.0 (partially implemented; completion blocked)
55	KIP-470: TopologyTestDriver test input and output usability improvements	2.4.0
56	KIP-467: Augment ProduceResponse error messaging for specific culprit records	2.4.0 (partially implemented) / 2.5.0
57	KIP-466: Add support for List<T> serialization and deserialization	2.4.0 (WIP)
58	KIP-465: Add Consolidated Connector Endpoint to Connect REST API	2.3.0
59	KIP-464: Defaults for AdminClient#createTopic	2.4.0
60	KIP-462: Use local thread id for KStreams	2.3.0
61	KIP-461: Improve Replica Fetcher behavior at handling partition failure	2.3.0
62	KIP-460: Admin Leader Election RPC	2.4.0
63	KIP-458: Connector Client Config Override Policy	2.3.0
64	KIP-455: Create an Administrative API for Replica Reassignment	2.4.0 (API/broker logic) 2.5.0 (tools changes)
65	KIP-454: Expansion of the ConnectClusterState interface	2.3.0
66	KIP-453: Add close() method to RocksDBConfigSetter	2.3.0
67	KIP-449: Add connector contexts to log messages in Connect workers	2.3.0
68	KIP-447: Producer scalability for exactly once semantics	2.5.0 (partially implemented) / 2.6.0 (WIP)
69	KIP-446: Add changelog topic configuration to KTable suppress	2.6.0
70	KIP-445: In-memory Session Store	2.3.0
71	KIP-444: Augment metrics for Kafka Streams	2.4.0 (partially implemented) / 2.5.0 (WIP)
72	KIP-443: Return to default segment.ms and segment.index.bytes in Streams repartition topics	2.3.0
73	KIP-442: Return to default max poll interval in Streams	2.3.0
74	KIP-441: Smooth Scaling Out for Kafka Streams	2.6.0 (WIP)
75	KIP-440: Extend Connect Converter to support headers	2.4.0
76	KIP-436: Add a metric indicating start time	2.3.0
77	KIP-434: Add Replica Fetcher and Log Cleaner Count Metrics	2.4.0 (WIP)
78	KIP-430 - Return Authorized Operations in Describe Responses	2.3.0
79	KIP-429: Kafka Consumer Incremental Rebalance Protocol	2.4.0 (WIP)
80	KIP-428: Add in-memory window store	2.3.0
81	KIP-427: Add AtMinIsr topic partition category (new metric & TopicCommand option)	2.3.0
82	KIP-425: Add some Log4J Kafka Appender Properties for Producing to Secured Brokers	2.3.0
83	KIP-421: Support resolving externalized secrets in AbstractConfig	2.3.0
84	KIP-420: Add Single Value Fetch in Session Stores	2.2.0
85	KIP-417: Allow JmxTool to connect to a secured RMI port	2.3.0
86	KIP-415: Incremental Cooperative Rebalancing in Kafka Connect	2.3.0
87	KIP-414: Expose Embedded ClientIds in Kafka Streams	2.2.0
88	KIP-412: Extend Admin API to support dynamic application log levels	2.4.0
89	KIP-411: Make default Kafka Connect worker task client IDs distinct	2.3.0
90	KIP-402: Improve fairness in SocketServer processors	2.2.0 (partially implemented) / 2.3.0
91	KIP-401: TransformerSupplier/ProcessorSupplier StateStore connecting	2.5.0 (WIP)
92	KIP-399: Extend ProductionExceptionHandler to cover serialization exceptions	2.5.0
93	KIP-396: Add Commit/List Offsets Operations to AdminClient	2.5.0 (WIP)
94	KIP-394: Require member.id for initial join group request	2.2.0
95	KIP-393: Time windowed serde to properly deserialize changelog input topic	2.2.0
96	KIP-392: Allow consumers to fetch from closest replica	2.4.0
97	KIP-389: Introduce a configurable consumer group size limit	2.2.0
98	KIP-386: Standardize on Min/Avg/Max metrics' default value	2.2.0
99	KIP-382: MirrorMaker 2.0	2.4.0
100	KIP-379: Multiple Consumer Group Management	2.4.0
101	KIP-380: Detect outdated control requests and bounced brokers using broker generation	2.2.0
102	KIP-377: TopicCommand to use AdminClient	2.2.0
103	KIP-376: Implement AutoClosable on appropriate classes that want to be used in a try-with-resource statement	2.2.0
104	KIP-374: Add '--help' option to all available Kafka CLI commands	2.2.0
105	KIP-372: Naming Repartition Topics for Joins and Grouping	2.1.0
106	KIP-371: Add a configuration to build custom SSL principal name	2.2.0
107	KIP-369: Alternative Partitioner to Support "Always Round-Robin" Selection	2.4.0
108	KIP 368: Allow SASL Connections to Periodically Re-Authenticate	2.2.0
109	KIP-367 Introduce close(Duration) to Producer and AdminClient instead of close(long, TimeUnit)	2.2.0
110	KIP-366: Make FunctionConversions deprecated	2.1.0
111	KIP-365: Materialized, Serialized, Joined, Consumed and Produced with implicit Serde	2.1.0
112	K IP-361: Add Consumer Configuration to Disable Auto Topic Creation	2.3.0
113	KIP-360: Improve reliability of idempotent/transactional producer	2.4.0 (partially implemented) / 2.5.0
114	KIP-359: Verify leader epoch in produce requests	2.4.0 (WIP)
115	KIP-358: Migrate Streams API to Duration instead of long ms times	2.1.0
116	KIP-357: Add support to list ACLs per principal	2.1.0
117	KIP-356: Add withCachingDisabled() to StoreBuilder	2.1.0
118	KIP-354: Add a Maximum Log Compaction Lag	2.3.0
119	KIP-353: Improve Kafka Streams Timestamp Synchronization	2.1.0
120	KIP-352: Distinguish URPs caused by reassignment	2.5.0
121	KIP-351: Add --under-min-isr option to describe topics command	2.3.0
122	KIP-346: Improve LogCleaner behavior on error	2.1.0
123	KIP-345: Introduce static membership protocol to reduce consumer rebalances	2.4.0
124	KIP-342 Add support for custom SASL extensions in OAuthBearer authentication	2.1.0
125	KIP-341: Update Sticky Assignor's User Data Protocol	2.3.0
126	KIP-340: Allow kafka-reassign-partitions.sh and kafka-log-dirs.sh to take admin client property file	2.1.0
127	KIP-339: Create a new IncrementalAlterConfigs API	2.3.0
128	KIP-338 Support to exclude the internal topics in kafka-topics.sh command	2.1.0
129	KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer	2.1.0
130	KIP-332: Update AclCommand to use AdminClient API	2.1.0
131	KIP-331 Add default implementation to close() and configure() for Serializer, Deserializer and Serde	2.3.0
132	KIP-330: Add retentionPeriod in SessionBytesStoreSupplier	2.1.0
133	KIP-328: Ability to suppress updates for KTables	2.1.0 (partially implemented; inactive)
134	KIP-324: Add method to get metrics() in AdminClient	2.1.0
135	KIP-322: Return new error code for DeleteTopics API when topic deletion disabled.	2.1.0
136	KIP-321: Update TopologyDescription to better represent Source and Sink Nodes	2.1.0
137	KIP-320: Allow fetchers to detect and handle log truncation	2.1.0 (partially implemented) / 2.2.0
138	KIP-319: Replace segments with segmentInterval in WindowBytesStoreSupplier	2.1.0
139	KIP-313: Add KStream.flatTransform and KStream.flatTransformValues	2.2.0 (partially implemented) / 2.3.0
140	KIP-312 Add Overloaded StreamsBuilder Build Method to Accept java.util.Properties	2.1.0
141	KIP-309: Add toUpperCase support to sasl.kerberos.principal.to.local rule	2.4.0
142	KIP-308: Support dynamic update of max.connections.per.ip/max.connections.per.ip.overrides	2.1.0
143	KIP-307: Allow to define custom processor names with KStreams DSL	2.3.0 (partially implemented) / 2.4.0
144	KIP-306: Configuration for Delaying Response to Failed Authentication	2.1.0
145	KIP-305: Add Connect primitive number converters	2.0.0
146	KIP-303: Add Dynamic Routing in Streams Sink	2.0.0
147	KIP-302 - Enable Kafka clients to use all DNS resolved IP addresses	2.1.0
148	KIP-300: Add Windowed KTable API in StreamsBuilder	3.0.0 (not implemented; inactive)
149	KIP-298: Error Handling in Connect	2.0.0
150	KIP-297: Externalizing Secrets for Connect Configurations	2.0.0
151	KIP-295 Add Streams Configuration Allowing for Optional Topology Optimization	2.0.0
152	KIP-294 - Enable TLS hostname verification by default	2.0.0
153	KIP-291: Separating controller connections and requests from the data plane	Accepted
154	KIP-292: Add transformValues() method to KTable	2.0.0
155	KIP-290: Support for Prefixed ACLs	2.0.0
156	KIP-289: Improve the default group id behavior in KafkaConsumer	2.2.0
157	KIP-285: Connect Rest Extension Plugin	2.0.0
158	KIP-284: Set default retention ms for Streams repartition topics to Long.MAX_VALUE	2.0.0
159	KIP-283: Efficient Memory Usage for Down-Conversion	2.0.0
160	KIP-282: Add the listener name to the authentication context	2.0.0
161	KIP-281: ConsumerPerformance: Increase Polling Loop Timeout and Make It Reachable by the End User	2.0.0
162	KIP-280: Enhanced log compaction	2.5.0 (WIP)
163	KIP-279: Fix log divergence between leader and follower after fast leader fail over	2.0.0
164	KIP-278 - Add version option to Kafka's commands	2.0.0
165	KIP-277 - Fine Grained ACL for CreateTopics API	2.0.0
166	KIP-276 - Add StreamsConfig prefix for different consumers	2.0.0
167	KIP-274: Kafka Streams Skipped Records Metrics	2.0.0
168	KIP-272: Add API version tag to broker's RequestsPerSec metric	2.0.0
169	KIP-270 - A Scala Wrapper Library for Kafka Streams	2.0.0
170	KIP-268: Simplify Kafka Streams Rebalance Metadata Upgrade	2.0.0
171	KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils	2.0.0
172	KIP-266: Fix consumer indefinite blocking behavior	2.0.0
173	KIP-265: Make Windowed Serde to public APIs	2.0.0
174	KIP-261: Add Single Value Fetch in Window Stores	2.0.0
175	KIP-258: Allow to Store Record Timestamps in RocksDB	2.3.0 (partially implemented; inactive)
176	KIP-257 - Configurable Quota Management	2.0.0
177	KIP-255: OAuth Authentication via SASL/OAUTHBEARER	2.0.0
178	KIP-251: Allow timestamp manipulation in Processor API	2.0.0
179	KIP-249: Add Delegation Token Operations to KafkaAdminClient	2.0.0
180	KIP-247: Add public test utils for Kafka Streams	1.1.0
181	KIP-245: Use Properties instead of StreamsConfig in KafkaStreams constructor	2.0.0
182	KIP-244: Add Record Header support to Kafka Streams Processor API	2.0.0
183	KIP-243: Make ProducerConfig and ConsumerConfig constructors public	1.1.0
184	KIP-239 Add queryableStoreName() to GlobalKTable	1.1.0
185	KIP-238: Expose Kafka cluster ID in Connect REST API	1.1.0
186	KIP-237: More Controller Health Metrics	2.0.0
187	KIP-235: Add DNS alias support for secured connection	2.1.0
188	KIP-233: Simplify StreamsBuilder#addGlobalStore	1.1.0
189	KIP-231: Improve the Required ACL of ListGroups API	2.1.0
190	KIP-229: DeleteGroups API	1.1.0
191	KIP-227 - Introduce Incremental FetchRequests to Increase Partition Scalability	1.1.0
192	KIP-226 - Dynamic Broker Configuration	1.1.0
193	KIP-225 - Use tags for consumer “records.lag” metrics	1.1.0
194	KIP-224: Add configuration parameter `retries` to Streams API	1.1.0
195	KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer	2.0.0
196	KIP-222 - Add Consumer Group operations to Admin API	2.0.0
197	KIP-221: Enhance DSL with Connecting Topic Creation and Repartition Hint	2.6.0
198	KIP-220: Add AdminClient into Kafka Streams' ClientSupplier	1.1.0
199	KIP-219 - Improve quota communication	2.0.0
200	KIP-218: Make KafkaFuture.Function java 8 lambda compatible	1.1.0
201	KIP-216: IQ should throw different exceptions for different errors	2.5.0
202	KIP-215: Add topic regex support for Connect sinks	1.1.0
203	KIP-214: Add zookeeper.max.in.flight.requests config to the broker	1.1.0
204	KIP-213 Support non-key joining in KTable	2.4.0 (WIP)
205	KIP-212: Enforce set of legal characters for connector names	1.1.0
206	KIP-211: Revise Expiration Semantics of Consumer Group Offsets	2.1.0
207	KIP-210 - Provide for custom error handling when Kafka Streams fails to produce	1.1.0
208	KIP-208: Add SSL support to Kafka Connect REST interface	1.1.0
209	KIP-207: Offsets returned by ListOffsetsResponse should be monotonically increasing even during a partition leader change	2.2.0
210	KIP-206: Add support for UUID serialization and deserialization	2.1.0
211	KIP-205: Add all() and range() API to ReadOnlyWindowStore	1.1.0
212	KIP-204 : Adding records deletion operation to the new Admin Client API	1.1.0
213	KIP-203: Add toLowerCase support to sasl.kerberos.principal.to.local rule	1.1.0
214	KIP-202: Move merge() from StreamsBuilder to KStream	1.0.0
215	KIP-198: Remove ZK dependency from Streams Reset Tool	1.0.0
216	KIP-197 Connect REST API should include the connector type when describing a connector	1.0.0
217	KIP-196: Add metrics to Kafka Connect framework	1.0.0
218	KIP-195: AdminClient.createPartitions	1.0.0
219	KIP-192 : Provide cleaner semantics when idempotence is enabled	1.0.0
220	KIP-191: KafkaConsumer.subscribe() overload that takes just Pattern	1.0.0
221	KIP-190: Handle client-ids consistently between clients and brokers	1.0.0
222	KIP-189: Improve principal builder interface and add support for SASL	1.0.0
223	KIP-188 - Add new metrics to support health checks	1.0.0
224	KIP-187 - Add cumulative count metric for all Kafka rate metrics	1.0.0
225	KIP-186: Increase offsets retention default to 7 days	2.0.0
226	KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient	2.2.0
227	KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines	1.0.0
228	KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress	1.1.0
229	KIP-177: Consumer perf tool should count rebalance time	1.0.0
230	KIP-176: Remove deprecated new-consumer option for tools	2.0.0
231	KIP-175: Additional '--describe' views for ConsumerGroupCommand	1.1.0
232	KIP-174 - Deprecate and remove internal converter configs in WorkerConfig	2.0.0
233	KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs	1.0.0
234	KIP-171 - Extend Consumer Group Reset Offset for Stream Application	1.1.0
235	KIP-168: Add GlobalTopicCount and GlobalPartitionCount metric per cluster	1.0.0
236	KIP-167: Add interface for the state store restoration process	1.0.0
237	KIP-164 - Add UnderMinIsrPartitionCount and per-partition UnderMinIsr metrics	1.0.0
238	KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch	1.0.0
239	KIP-162: Enable topic deletion by default	1.0.0
240	KIP-161: streams deserialization exception handlers	1.0.0
241	KIP-160: Augment KStream.print(), KStream.writeAsText() to allow users pass in extra parameters in the printed string	1.0.0
242	KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics	2.5.0 (WIP)
243	KIP-157 - Add consumer config options to streams reset tool	1.0.0
244	KIP-156 Add option "dry run" to Streams application reset tool	0.11.0.0
245	KIP-155 - Add range scan for windowed state stores	0.11.0.0
246	KIP-154 Add Kafka Connect configuration properties for creating internal topics	0.11.0.0
247	KIP-153: Include only client traffic in BytesOutPerSec metric	0.11.0.0
248	KIP-152 - Improve diagnostics for SASL authentication failures	1.0.0
249	KIP-151 Expose Connector type in REST API	0.11.0.0
250	KIP-150 - Kafka-Streams Cogroup	2.5.0 (WIP)
251	KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner	1.1.0 (partially implemented; inactive)
252	KIP-146 - Classloading Isolation in Connect	0.11.0.0
253	KIP-145 - Expose Record Headers in Kafka Connect	1.1.0
254	KIP-144: Exponential backoff for broker reconnect attempts	0.11.0.0
255	KIP-143: Controller Health Metrics	0.11.0.0
256	KIP-140: Add administrative RPCs for adding, deleting, and listing ACLs	0.11.0.0
257	KIP-138: Change punctuate semantics	1.0.0
258	KIP-137: Enhance TopicCommand --describe to show topics marked for deletion	0.11.0.0
259	KIP-136: Add Listener name to SelectorMetrics tags	0.11.0.0
260	KIP-134: Delay initial consumer group rebalance	0.11.0.0
261	KIP-133: Describe and Alter Configs Admin APIs	0.11.0.0
262	KIP-130: Expose states of active tasks to KafkaStreams public API	1.0.0
263	KIP-129: Streams Exactly-Once Semantics	0.11.0.0
264	KIP-128: Add ByteArrayConverter for Kafka Connect	0.11.0.0
265	KIP-126 - Allow KafkaProducer to split and resend oversized batches.	0.11.0.0
266	KIP-124 - Request rate quotas	0.11.0.0
267	KIP-123: Allow per stream/table timestamp extractor	0.11.0.0
268	KIP-122: Add Reset Consumer Group Offsets tooling	0.11.0.0
269	KIP-121: Add KStream peek method	0.11.0.0
270	KIP-120: Cleanup Kafka Streams builder API	1.0.0
271	KIP-119: Drop Support for Scala 2.10 in Kafka 0.11	0.11.0.0
272	KIP-118: Drop Support for Java 7	2.0.0
273	KIP-117: Add a public AdminClient API for Kafka admin operations	0.11.0.0
274	KIP-115: Enforce offsets.topic.replication.factor upon __consumer_offsets auto topic creation	0.11.0.0
275	KIP-114: KTable state stores and improved semantics	0.11.0.0
276	KIP-113: Support replicas movement between log directories	1.1.0
277	KIP-112: Handle disk failure for JBOD	1.0.0
278	KIP-110: Add Codec for ZStandard Compression	2.1.0
279	KIP-109: Old Consumer Deprecation	0.11.0.0
280	KIP-108: Create Topic Policy	0.10.2.0
281	KIP-107: Add deleteRecordsBefore() API in AdminClient	0.11.0.0
282	KIP-106 - Change Default unclean.leader.election.enabled from True to False	0.11.0.0
283	KIP-105: Addition of Record Level for Sensors	0.10.2.0
284	KIP-104: Granular Sensors for Streams	0.10.2.0
285	KIP-103: Separation of Internal and External traffic	0.10.2.0
286	KIP-102 - Add close with timeout for consumers	0.10.2.0
287	KIP-101 - Alter Replication Protocol to use Leader Epoch rather than High Watermark for Truncation	0.11.0.0
288	KIP-100 - Relax Type constraints in Kafka Streams API	0.10.2.0
289	KIP-99: Add Global Tables to Kafka Streams	0.10.2.0
290	KIP-98 - Exactly Once Delivery and Transactional Messaging	0.11.0.0
291	KIP-97: Improved Kafka Client RPC Compatibility Policy	0.10.2.0
292	KIP-96 - Add per partition metrics for in-sync and assigned replica count	0.10.2.0
293	KIP-94 Session Windows	0.10.2.0
294	KIP-93: Improve invalid timestamp handling in Kafka Streams	0.10.2.0
295	KIP-92 - Add per partition lag metrics to KafkaConsumer	0.10.2.0
296	KIP-91 Provide Intuitive User Timeouts in The Producer	2.1.0
297	KIP-90 - Remove zkClient dependency from Streams	0.10.2.0
298	KIP-89: Allow sink connectors to decouple flush and offset commit	0.10.2.0
299	KIP-88: OffsetFetch Protocol Update	0.10.2.0
300	KIP-86: Configurable SASL callback handlers	2.0.0
301	KIP-85: Dynamic JAAS configuration for Kafka clients	0.10.2.0
302	KIP-84: Support SASL SCRAM mechanisms	0.10.2.0
303	KIP-82 - Add Record Headers	0.11.0.0
304	KIP-81: Bound Fetch memory usage in the consumer	2.6.0 (WIP)
305	KIP-79 - ListOffsetRequest/ListOffsetResponse v1 and add timestamp search methods to the new consumer	0.10.1.0
306	KIP-78: Cluster Id	0.10.1.0
307	KIP-77: Improve Kafka Streams Join Semantics	0.10.2.0
308	KIP-75 - Add per-connector Converters	0.10.1.0
309	KIP-74: Add Fetch Response Size Limit in Bytes	0.10.1.0
310	KIP-73: Replication Quotas	0.10.1.0
311	KIP-72: Allow putting a bound on memory consumed by Incoming request	1.0.0
312	KIP-71: Enable log compaction and deletion to co-exist	0.10.1.0
313	KIP-70: Revise Partition Assignment Semantics on New Consumer's Subscription Change	0.10.1.0
314	KIP-67: Queryable state for Kafka Streams	0.10.1.0
315	KIP-66: Single Message Transforms for Kafka Connect	0.10.2.0 / 0.11.0.0
316	KIP-65: Expose timestamps to Connect	0.10.1.0
317	KIP-63: Unify store and downstream caching in streams	0.10.1.0
318	KIP-62: Allow consumer to send heartbeats from a background thread	0.10.1.0
319	KIP-60 - Make Java client classloading more flexible	0.10.1.0
320	KIP-58 - Make Log Compaction Point Configurable	0.10.1.0
321	KIP-57 - Interoperable LZ4 Framing	0.10.0.0
322	KIP-56: Allow cross origin HTTP requests on all HTTP methods	0.10.0.0
323	KIP-55: Secure Quotas for Authenticated Users	0.10.1.0
324	KIP-54: Sticky Partition Assignment Strategy	0.11.0.0
325	KIP-52: Connector Control APIs	0.10.0.0
326	KIP-51 - List Connectors REST API	0.10.0.0
327	KIP-48 Delegation token support for Kafka	1.1.0
328	KIP-45 - Standardize all client sequence interaction on j.u.Collection.	0.10.0.0
329	KIP-43: Kafka SASL enhancements	0.10.0.0
330	KIP-42: Add Producer and Consumer Interceptors	0.10.0.0
331	KIP-41: Consumer Max Records	0.10.0.0
332	KIP-40: ListGroups and DescribeGroup	0.9.0.0
333	KIP-38: ZooKeeper Authentication	0.9.0.0
334	KIP-36 - Rack aware replica assignment	0.10.0.0
335	KIP-35 - Retrieving protocol version	0.10.0.0
336	KIP-33 - Add a time based log index	0.10.1.0
337	KIP-32 - Add timestamps to Kafka message	0.10.0.0
338	KIP-31 - Move to relative offsets in compressed message sets	0.10.0.0
339	KIP-28 - Add a processor client	0.10.0.0
340	KIP-26 - Add Kafka Connect framework for data import/export	0.9.0.0
341	KIP-25 - System test improvements	0.9.0.0
342	KIP-22 - Expose a Partitioner interface in the new producer	0.9.0.0
343	KIP-21 - Dynamic Configuration	0.9.0.0 (WIP)
344	KIP-20 Enable log preallocate to improve consume performance under windows and some old Linux file system	0.9.0.0
345	KIP-19 - Add a request timeout to NetworkClient	0.9.0.0
346	KIP-16 - Automated Replica Lag Tuning	0.9.0.0
347	KIP-15 - Add a close method with a timeout in the producer	0.9.0.0
348	KIP-13 - Quota Design	0.9.0.0
349	KIP-12 - Kafka Sasl/Kerberos and SSL implementation	0.9.0.0
350	KIP-11 - Kafka Authorizer design	0.9.0.0
351	KIP-8 - Add a flush method to the producer API	0.9.0.0
352	KIP-4 - Metadata Protocol Changes	0.10.0.0
353	KIP-4 - Command line and centralized administrative operations	0.9.0.0, 0.10.0.0, 0.10.1.0
354	KIP-3 - Mirror Maker Enhancement	0.9.0.0
355	KIP-2 - Refactor brokers to allow listening on multiple ports and IPs	0.9.0.0
356	KIP-1 - Remove support of request.required.acks	0.9.0.0

KIPs under discussion

Please insert new rows in sorted order (ascending by KIP number). Please move the adopted/accepted KIPs to the "Adopted KIPs" table above.

KIP	Comment
KIP-59: Proposal for a kafka broker command	Sent emails to Dev discussion group. Work tracked under KAFKA-3663.
KIP-125: ZookeeperConsumerConnector to KafkaConsumer Migration and Rollback
KIP-131 - Add access to OffsetStorageReader from SourceConnector
KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user
KIP 141 - ProducerRecord & SourceRecord: Add timestamp constructors
KIP-142: Add ListTopicsRequest to efficiently list all the topics in a cluster
KIP-148: Add a connect timeout for client
KIP-166 - Add a tool to make amounts of replicas and leaders on brokers balanced
KIP-169 - Lag-Aware Partition Assignment Strategy
KIP-178: Size-based log directory selection strategy
KIP-185: Make exactly once in order delivery the default producer setting
KIP-193: Add SchemaBuilder.from(Schema)
KIP-199: Add Kafka Connect offset tool
KIP-201: Rationalising Policy interfaces
KIP-209: C onnection String Support
KIP-217: Expose a timeout to allow an expired ZK session to be re-created
KIP-228 Negative record timestamp support	Voting in progress
KIP-234: add support for getting topic defaults from AdminClient
KIP-236: Interruptible Partition Reassignment	Discussion
KIP-240: AdminClient.listReassignments() AdminClient.describeReassignments()
KIP-242: Mask password in Kafka Connect Rest API response
KIP-250 Add Support for Quorum-based Producer Acknowledgment
KIP-252 - Extend ACLs to allow filtering based on ip ranges and subnets
KIP-253: Support in-order message delivery with partition expansion	Discussion
KIP-254: JsonConverter Exception Handeling
KIP-264: Add a consumer metric to record raw fetch size	Voting in progress
KIP-271: Add NetworkClient redirector	Discussion
KIP-273: Kafka to support using ETCD beside Zookeeper	Discussion
KIP-275 - Indicate "isClosing" in the SinkTaskContext	Voting in progress
KIP-293 Add new metrics for consumer/replication fetch requests	Voting in progress
KIP-296: Add connector level configurability for producer/consumer client configs	Discussion
KIP-301: Schema Inferencing for JsonConverter	Discussion
KIP-304: Connect runtime mode improvements for container platforms	Discussion
KIP-315: Stream Join Sticky Assignor	Discussion
KIP-316: Command-line overrides for ConnectDistributed worker properties	Discussion
KIP-317: Add end-to-end data encryption functionality to Apache Kafka	Discussion
KIP-325: Extend Consumer Group Command to Show Beginning Offsets	Voting in Progress345
KIP-333: Add faster mode of rebalancing	Discussion
KIP-334 - Include partitions in exceptions raised during consumer record deserialization/validation
KIP-347: Enable batching in FindCoordinatorRequest	Discussion
KIP-348 Eliminate null from SourceTask#poll()
KIP-350: Allow kafka-topics.sh to take brokerid as parameter to show partitions associated with it
KIP-356: Add KafkaConsumer fetch-error-rate and fetch-error-total metrics	Discussion
KIP-363: Allow performance tools to print final results to output file	Discussion
KIP-370: Remove Orphan Partitions	Discussion
KIP-373: Allow users to create delegation tokens for other users	Accepted
KIP-375: Kafka Clients - make Metadata#TOPIC_EXPIRY_MS configurable	Discussion
KIP-381: Connect: Tell about records that had their offsets flushed in callback	Voting in progress (restarted 18th January 2019, due to no votes in first attempt)
KIP-383: Pluggable interface for SSL Factory	Voting in progress
KIP-384: Add config for incompatible changes to persistent metadata	Discussion
KIP-385: Avoid throwing away prefetched data	Discussion
KIP-387: Fair Message Consumption Across Partitions in KafkaConsumer	Discussion
KIP-388: Add observer interface to record request and response	Discussion
KIP-390: Allow fine-grained configuration for compression	Discussion, JIRA exists with pull-request
KIP-395: Encypt-then-MAC Delegation token metadata
KIP-397: Add methods to override fetch offsets based on timestamp	Discussion
KIP-400: Improve exit status in case of errors in ConsoleProducer	Discussion
KIP-403: Increase ProducerPerformance precision by using nanoTime	Draft
KIP-405: Kafka Tiered Storage	Discussion
KIP-407: Kafka Connect support override worker kafka api configuration with connector configuration that post by rest api
KIP-409: Allow creating under-replicated topics and partitions
KIP-410: Add metric for request handler thread pool utilization by request type	Discussion
KIP-416: Notify SourceTask of ACK'd offsets, metadata	Discussion
KIP-419: Safely notify Kafka Connect SourceTask is stopped	Voting in progress, JIRA exists with pull request
KIP-422: Add support for client quota configuration in the Kafka Admin Client	Discussion, JIRA exists with pull-request
KIP-423: Add JoinReason to Consumer Join Group Protocol	Discussion
KIP-424: Allow suppression of intermediate events based on wall clock time	Discussion
KIP-426: Persist Broker Id to Zookeeper	Discussion
KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter	Voting in progress, JIRA exists with pull-request
KIP-435: Internal Partition Reassignment Batching	Discussion
KIP-437: Custom replacement for MaskField SMT	Voting in progress, JIRA exists with pull-request
KIP-438: Expose task, connector IDs in Connect API	Discussion
KIP-439: Cleanup built-in Store interfaces	Discussion
KIP-448: Add State Stores Unit Test Support to Kafka Streams Test Utils	Discussion
KIP-452: Tool to view cluster status	Discussion, JIRA exists
KIP-468: Avoid decompression of record when validate record at server in the scene of inPlaceAssignment .	Under discussion
KIP-473: Enable KafkaLog4JAppender to use SASL Authentication Callback Handlers	Under Discussion
KIP-477: Add PATCH method for connector config in Connect REST API	Under Discussion
KIP-486: Support custom way to load KeyStore and TrustStore	Under Discussion
KIP-487: Automatic Topic Creation on Producer	Under Discussion
KIP-489: Kafka Consumer Record Latency Metric	Under Discussion
KIP-490: New metric to count offsets expired without being consumed by a consumer group	Under discussion
KIP-491: Preferred Leader Deprioritized List (Temporary Blacklist)	Under discussion
KIP-494: Connect REST Endpoint to get Transformations (SMTs)
KIP-498: Add client-side configuration for maximum response size to protect against OOM	Sent emails to Dev discussion group. Work tracked under KAFKA-4090.
KIP-499 - Unify connection name flag for command line tool	Under discussion
KIP-502: Connect SinkTask.put(...) to specify ArrayList<SinkRecord> in Signature
KIP-505: Add new public method to only update assignment metadata in consumer	Under discussion
KIP-506: Allow setting SCRAM password via Admin interface	Under discussion
KIP-509: Rebalance and restart Producers	Under discussion
KIP-510: Metrics library upgrade	Under discussion
KIP-512: Adding headers to RecordMetaData	Sent mail to dev discussion group
KIP-513: Distinguish between Key and Value serdes in scala wrapper library for kafka streams	Sent mail to dev discussion group
KIP-516: Topic Identifiers & Topic Deletion State Improvements	Under discussion
KIP-519: Make SSL context/engine configuration extensible	Voting
KIP-536: Propagate broker start time to Admin API	Under discussion
KIP-539: Add mechanism to flush records out in low volume suppression buffers	Under discussion.
KIP-540: Implement per key stream time tracking	Draft (In Progress)
KIP-542: Partition Reassignment Throttling	Draft
KIP-545: support automated consumer offset sync across clusters in MM 2.0	Voting in progress
KIP-547: Extend ConsumerInterceptor to allow modification of Consumer Commits	Under Discussion
KIP-548 Add Option to enforce rack-aware custom partition reassignment execution	Under Discussion
KIP-549: Surface Consumer's Metadata in KafkaAdminClient#describeConsumerGroups	Under Discussion
KIP-550: Mechanism to Delete Stray Partitions on Broker	Draft
KIP-552: Suppress "unused config" warning	Under Discussion
KIP-560: Auto infer external topic partitions in stream reset tool	Under Discussion
KIP-561: Regex Support for ConsumerGroupCommand	Under Discussion
KIP-563: Add 'tail -n' feature for ConsoleConsumer	Under Discussion
KIP-564: Add new cached authorizer:change the dim of cache	Under Discussion
KIP-565: Using AclCommand,avoid call the global method loadcache in SimpleAclAuthorizer	Under Discussion
KIP-566: Add rebalance callbacks to ConsumerInterceptor	Under Discussion
KIP-567: Kafka Cluster Audit	Under Discussion
KIP-569: DescribeConfigsResponse - Update the schema to include additional metadata information of the field	Under Discussion
KIP-571: Add option to force remove members in StreamsResetter	Voting
KIP-572: Improve timeouts and retries in Kafka Streams	Under Discussion
KIP-573: Enable TLSv1.3 by default	Under Discussion
KIP-574: CLI Dynamic Configuration with file input	Voting in progress
KIP-575: build a Kafka-Exporter by Java	Under Discussion
KIP-576: Support dynamic update of more broker configs related to replication	Under Discussion
KIP-577: Allow HTTP Response Headers Configured for Kafka Connect	Under Discussion
KIP-580: Exponential Backoff for Kafka Clients	Under Discussion
KIP-581: Value of optional null field which has default value	Under Discussion

Dormant/inactive KIPs

Please insert new rows in sorted order (ascending by KIP number).

KIP	Comment
KIP-6 - New reassignment partition logic for rebalancing	Needs more detail
KIP-10 - Running Producer, Consumers and Brokers on Mesos
KIP-14 - Tools standardization
KIP-17 - Add HighwaterMarkOffset to OffsetFetchResponse
KIP-23 - Add JSON/CSV output and looping options to ConsumerGroupCommand
KIP-27 - Conditional Publish
KIP-30 - Allow for brokers to have plug-able consensus and meta data storage sub systems
KIP-37 - Add Namespaces to Kafka
KIP-39: Pinning controller to broker
KIP-44 - Allow Kafka to have a customized security protocol
KIP-46 - Self Healing
KIP-47 - Add timestamp-based log deletion policy
KIP-49 - Fair Partition Assignment Strategy
KIP-53 - Add custom policies for reconnect attempts to NetworkdClient
KIP-59: Proposal for a kafka broker command
KIP-64 -Allow underlying distributed filesystem to take over replication depending on configuration
KIP-68 Add a consumed log retention before log retention
KIP-69 - Kafka Schema Registry	Draft
KIP-76 Enable getting password from executable rather than passing as plaintext in config files
KIP-87 - Add Compaction Tombstone Flag	Possibly displaced by KIP-82
KIP-95: Incremental Batch Processing for Kafka Streams
KIP-159: Introducing Rich functions to Streams
KIP-260: add primary join operation for Stream-Stream join (WIP)	Draft
KIP-269: Substitution Within Configuration Values
KIP-311: Async processing with dynamic scheduling in Kafka Streams
KIP-314: KTable to GlobalKTable Bi-directional Join
KIP-326: Schedulable KTable as Graph source
KIP-349: Priorities for Source Topics	Functionality can be realized with existing API although at a lower level.
KIP-335: Consider configurations for KafkaStreams
K IP-362: Support Dynamic Session Window
KIP-378: Enable Dependency Injection for Kafka Streams handlers
KIP-406: GlobalStreamThread should honor custom offset policy.
KIP-408: Add asynchronous processing to Kafka Streams
KIP-418: A method-chaining way to branch KStream
KIP-450: Sliding Window Aggregations in the DSL
KIP-457: Add DISCONNECTED status to Kafka Streams
KIP-459: Improve KafkaStreams#close
KIP-463: Auto-configure non-default Serdes passed alongside the TopologyBuilder
KIP-508: Make Suppression State Queriable

Discarded KIPs