This page describes a proposed Kafka Improvement Proposal (KIP) process for proposing a major change to Kafka.
...
Create from template | ||||||||
---|---|---|---|---|---|---|---|---|
|
...
Table of Contents |
---|
Getting Started
If this is your first time contributing:
- Sign up for the Developer mailing list
dev@kafka.apache.org
. The instructions to sign up are here: http://kafka.apache.org/contact
...
- Create a wiki ID (https://cwiki.apache.org/confluence/signup.action)
- Create a Jira ID (It's a different system than the wiki) using the ASF Self-serve Portal (https://selfserve.apache.org/jira-account.html)
- Send an email to the dev mailing list (
dev@kafka.apache.org)
containing your wiki ID and Jira ID requesting permissions to contribute to Apache Kafka.
Purpose
We
Table of Contents |
---|
Purpose
We want to make Kafka a core architectural component for users. We also support a large number of integrations with other tools, systems, and clients. Keeping this kind of usage health requires a high level of compatibility between releases — core architectural elements can't break compatibility or shift functionality from release to release. As a result each new major feature or public api has to be done in a way that we can stick with it going forward.
...
- Click
. Take the next available KIP number and give your proposal a descriptive heading. e.g. "KIP 42: Allow Infinite Retention With Bounded Disk Usage".Create from template templateName 54329345 templateId 54329345 title KIP-NEXT: Insert Title Here buttonLabel Create KIP - Update the next available KIP number below.
- Fill in the sections as described above
- Also add an entry to the table KIPs under discussion (for Streams API KIPs, please also add it to Kafka Streams sub page).
- Start a [DISCUSS] thread on the Apache mailing list. Please ensure that the subject of the thread is of the format [DISCUSS] KIP-{your KIP number} {your KIP heading} and the body contains a link to your new KIP. The discussion should happen on the mailing list, not on the wiki, since the wiki comment system doesn't work well for larger discussions. In the process of the discussion you may update the proposal. You should let people know the changes you are making. When you feel you have a finalized proposal
- Once the proposal is finalized call a [VOTE] to have the proposal adopted. These proposals are more serious than code changes and more serious even than release votes. The criteria for acceptance is lazy majority. The vote should remain open for at least 72 hours.
- Please update the KIP wiki page, and the index below, to reflect the current stage of the KIP after a vote. This acts as the permanent record indicating the result of the KIP (e.g., Accepted or Rejected). Also report the result of the KIP vote to the voting thread on the mailing list so the conclusion is clear.e
KIP round-up
Next KIP Number: 6651020
Use this number as the identifier for your KIP and increment this value.
...
Please insert new rows in sorted order (descending by KIP number).
KIP (please keep this sorted by KIP number) | Release |
---|---|
KIP- |
1013: Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7) | 4.0.0 |
KIP-1012: The need for a Kafka 3.8.x release | 3.8 |
.0 |
KIP- |
1007: Introduce Remote Storage Not Ready Exception | 3.8 |
.0 |
KIP- |
1005: Expose EarliestLocalOffset and TieredOffset | 3.8 |
.0 |
KIP- |
1004: Enforce tasks.max property in Kafka Connect | 3.8.0 |
KIP-1001: Add CurrentControllerId Metric | 3 |
.7.0 |
KIP- |
1000: List Client Metrics Configuration Resources | 3 |
.7.0 |
TBD (targeting 3.8.0/4.0.0 |
) |
.0 |
KIP- |
985: Add reverseRange and reverseAll query over kv-store in IQv2 | 3 |
.7.0 |
KIP- |
980: Allow creating connectors in a stopped state | TBD (targeting 3.7.0) |
TBD (targeting 3.7.0) |
3.7.0 | |
TBD (targeting 3.8.0) | |
KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect | 3.7 |
.0 |
KIP- |
975: Docker Image for Apache Kafka | TBD (targeting 3 |
.7.0 |
) |
KIP- |
974: Docker Image for GraalVM based Native Kafka Broker | TBD (targeting 3 |
.7.0 |
) |
KIP- |
970: Deprecate and remove Connect's redundant task configurations endpoint | 3. |
7.0 ( |
deprecation) |
4.0.0 ( |
removal) |
KIP- |
968: Support single-key_multi-timestamp Interactive Queries (IQv2) for Versioned State Stores | 3 |
.7.0 |
KIP- |
966: Eligible Leader Replicas | TBD |
KIP- |
963: Additional metrics in Tiered Storage | 3.7 |
.0 |
KIP- |
962: Relax non-null key requirement in Kafka Streams | 3 |
.7.0 |
KIP- |
960: Support single-key_single-timestamp Interactive Queries (IQv2) for Versioned State Stores | 3 |
.7.0 |
KIP- |
959: Add BooleanConverter to Kafka Connect |
3. |
7.0 |
KIP- |
954: expand default DSL store configuration to custom types | 3.7 |
.0 |
3.7 |
.0 |
KIP- |
.6.0 |
KIP- |
938: Add more metrics for measuring KRaft performance | TBD |
KIP-937: Improve Message Timestamp Validation | 3 |
.6.0 |
KIP- |
927: |
Improve the kafka-metadata-quorum output | 3 |
.6.0 |
KIP- |
925: Rack aware task assignment in Kafka Streams | 3.7.0 (completed) 3 |
.6.0 (partially implemented) |
KIP- |
923: Add A Grace Period to Stream Table Join | 3.6.0 |
3.7 |
.0 |
KIP- |
917: Additional custom metadata for remote log segment | 3.6.0 |
KIP- |
915: Txn and Group Coordinator Downgrade Foundation | 3 |
.5.0 |
KIP- |
914: DSL Processor Semantics for Versioned Stores | 3.5 |
.0 |
KIP- |
911: Add source tag to Mirror source metric | 3 |
.5.0 |
KIP- |
909: DNS Resolution Failure Should Not Fail the Clients | TBD |
KIP- |
907: Add |
Boolean Serde to public interface | 3.5.0 |
KIP- |
906: Tools migration guidelines | 3.5 |
.0 |
KIP- |
904: Kafka Streams - Guarantee subtractor is called before adder if key has not changed | 3 |
.5.0 |
KIP- |
903: Replicas with stale broker epoch should not be allowed to join the ISR | 3 |
.5.0 |
KIP- |
902: Upgrade Zookeeper to 3.8.2 | 3.6 |
.0 |
KIP- |
900: KRaft kafka-storage.sh API additions to support SCRAM for Kafka Brokers | 3 |
.5.0 |
KIP- |
898: Modernize Connect plugin discovery | 3.6 |
.0 |
KIP- |
896: Remove old client protocol API versions in Kafka 4.0 | 3.7.0 - new metric & request log attribute 4.0.0 - actual removal |
KIP-894: Use incrementalAlterConfig for syncing topic configurations | 3.5.0 |
KIP-893: The Kafka protocol should support nullable structs | 3 |
.5.0 |
KIP- |
892: Transactional Semantics for StateStores | TBD (targeting 3.7.0 |
) |
KIP- |
890: Transactions Server-Side Defense | TDB |
KIP- |
889: Versioned State Stores | 3.5 |
.0 |
KIP- |
887: Add |
ConfigProvider to make use of environment variables | 3 |
.5.0 |
KIP- |
884: Add config to configure KafkaClientSupplier in Kafka Streams | 3 |
.5.0 |
KIP- |
881: Rack-aware Partition Assignment for Kafka Consumers | 3 |
.4. |
0 (protocol) 3 |
.5.0 (assignors) |
KIP- |
878: Autoscaling for Stateless & Statically Partitioned Streams | TBD |
KIP- |
876: Time based cluster metadata snapshots | 3 |
.4.0 |
KIP- |
2.4.0 (partially implemented; completion blocked)
2.5.0 (tools changes)
2.2.0 (partially implemented)
/ 2.3.0
KIP-164 - Add UnderMinIsrPartitionCount and per-partition UnderMinIsr metrics
0.11.0.0
0.11.0.0
KIP-106 - Change Default unclean.leader.election.enabled from True to False
...
Please insert new rows in sorted order (ascending by KIP number). Please move the adopted/accepted KIPs to the "Adopted KIPs" table above.
...
KIP-185: Make exactly once in order delivery the default producer setting
...
Voting in progress (restarted 18th January 2019, due to no votes in first attempt)
...
Discussion
...
Discussion
...
...
...
Discussion
...
...
Under Discussion
...
KIP-560: Auto infer external topic partitions in stream reset tool
...
...
Under Discussion
...
...
...
...
KIPs under discussion
Please insert new rows in sorted order (ascending by KIP number). Please move the adopted/accepted KIPs to the "Adopted KIPs" table above.
Dormant/inactive KIPs
Please insert new rows in sorted order (ascending by KIP number).
Discarded KIPs
Please insert new rows in sorted order (ascending by KIP number).
Dormant/inactive KIPs
Please insert new rows in sorted order (ascending by KIP number).
...
KIP-269: Substitution Within Configuration Values
...
Functionality can be realized with existing API although at a lower level.
...
Discarded KIPs
Please insert new rows in sorted order (ascending by KIP number).
KIP | Comment | ||||||||
---|---|---|---|---|---|---|---|---|---|
KIP-5 - Broker Configuration Management | Superseded by KIP-21 | KIP-7 - Security - IP Filtering | |||||||
KIP-9 - SSL Support | Adopted via KAFKA-1690 but not via this KIP process | ||||||||
KIP-18 - JBOD Support | Superseded by KIP-112 and KIP-113 | KIP-24 - Remove ISR information from TopicMetadataRequest and add broker level metadata request | |||||||
KIP-29 - Add an IsrPropagateIntervalMs configuration to KafkaConfig | No longer need after KAFKA-2722 | ||||||||
KIP-34 Add Partitioner Change Listener to Partitioner Interface for Multiple Use Case | Per feedback it is better do this parition() method and avoid Thread Coordination etc. | ||||||||
KIP-50 - Move Authorizer to o.a.k.common package | Superseded by KIP-504 | KIP-80: Kafka Rest Server | |||||||
KIP-83 - Allow multiple SASL authenticated Java clients in a single JVM process | After KIP-85, no interface changes, reduced to KAFKA-4180 | ||||||||
KIP-111: Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker. | Covered by KIP-189 | ||||||||
KIP-116: Add State Store Checkpoint Interval Configuration | Currently not needed as checkpointing can be done on commit interval. | ||||||||
KIP-127: Pluggable JAAS LoginModule configuration for SSL | Similar functionality can be implemented using a custom PrincipalBuilder. | ||||||||
KIP-132: Augment KStream.print to allow extra parameters in the printed string | Duplicated by KIP-160 | ||||||||
KIP-147: Add missing type parameters to StateStoreSupplier factories and KGroupedStream/Table methods | Covered by KIP-182 | ||||||||
KIP-165: Extend Interactive Queries for return latest update timestamp per key | Covered by KIP-258 | ||||||||
KIP-170: Enhanced TopicCreatePolicy and introduction of TopicDeletePolicy | Superseded by KIP-201: Rationalising Policy interfaces | ||||||||
KIP 172: Add regular-expression topic support for sink connector | Superseded by KIP-215: Add topic regex support for Connect sinks | KIP-179: Change ReassignPartitionsCommand to use AdminClient | |||||||
KIP-184: Rename LogCleaner and related classes to LogCompactor | A lot of configuration change will cause more trouble and doesn't seems to be worth it | ||||||||
KIP 230: Name Windowing Joins | Covered by KIP-372: Naming Repartition Topics for Joins and Grouping | ||||||||
KIP-232: Detect outdated metadata using per-partition leaderEpoch field | Merged into KIP-320: Allow fetchers to detect and handle log truncation | ||||||||
KIP-246: Connect producers and consumers should inherit worker configs | Withdrawn because proposal could not be made backward compatible with existing behavior | ||||||||
KIP-248 - Create New ConfigCommand That Uses The New AdminClient | Discarded by author as it's partly covered in KIP-339 and KIP-422 | ||||||||
KIP-259: Improve Streams DSL Timestamp Propagation Semantics | Change did not require a KIP. | ||||||||
KIP-262: Metadata should include number of state stores for task | Rejected since metadata is no longer required. | ||||||||
KIP-263: Allow broker to skip sanity check of inactive segments on broker startup | Withdrawn because the solution we agreed on does not require interface change | KIP-275 - Indicate "isClosing" in the SinkTaskContext | |||||||
KIP-286: producer.send() should not block on metadata update | Withdrawn because the benefit of not having to wait for metadata is probably not worth the complexity added in producer. | ||||||||
KIP-288: [DISCARDED] Consumer.poll() timeout semantic change and new waitForAssignment method | Discarded in deference to KIP-266 | ||||||||
KIP-310: Add a Kafka Source Connector to Kafka Connect | Withdrawn in favor of KIP-382. | ||||||||
KIP-327: Add describe all topics API to AdminClient | Withdrawn in-favor of filter support in Metadata API and KIP-142 | ||||||||
KIP-344: The auto-generated client id should be passed to MetricsReporter | Bug fix so KIP not needed | ||||||||
KIP-364: Remove implicit Materialized, Consumed and Produced | Subsumed by KIP-365 | ||||||||
KIP-383: Pluggable interface for SSL Factory | Replaced by KIP-519 | ||||||||
KIP-398: Support reading trust store from classpath | Not enough interest in favour. Can be implemented with mechanism in
| ||||||||
KIP-404: Add Kafka Connect configuration parameter for disabling WADL output on OPTIONS request | Discarded, reported as a bug by
| ||||||||
A "security sanity check tool" is a more generic and appropriate solution. | |||||||||
KIP-451: Make TopologyTestDriver output iterable | Discarded in favor of KIP-456 | ||||||||
KIP-456: Helper classes to make it simpler to write test logic with TopologyTestDriver | Discarded in favor of KIP-470 | ||||||||
KIP-472: Add header to RecordContext/ProducerRecord | Discarded due to change being far more complex than necessary for a simple bug | ||||||||
KIP-485: Make topic optional when using through() operations in DSL | Moved to KIP-221 | ||||||||
KIP-486: Support custom way to load KeyStore and TrustStore | Discarded in favor of KIP-519 | ||||||||
KIP-522: Update BrokerApiVersionsCommand to use AdminClient | Discarded as this KIP is likely to be misused | ||||||||
KIP-615: add ConstrainedCooperativeStickyAssignor | Discarded in favor of modifying existing assignor | ||||||||
KIP-638: Deprecate DescribeLogDirsResponse.[LogDirInfo, ReplicaInfo] | Duplication of KIP-621
KIP Discussion Recordings
...