Versions Compared

Old Version 3

changes.mady.by.user sanghyeok an

Saved on

compared with

New Version Current

changes.mady.by.user sanghyeok an

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: Under Discussion

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: KAFKA-20029

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

This issue is not limited to __transaction_state :

  • __consumer_offsets: Expansion causes existing consumer groups to lose track of their committed offsets because the groupId mapping shifts to a new partition.Partition expansion of __consumer_offsets can cause group IDs to be mapped to different partitions across brokers while metadata is converging. As a result, offset commits, offset fetches, and group coordination operations may fail in a non-deterministic manner depending on which broker handles the request. During this window, it may be impossible to reliably read and write committed offsets, and some groups may become effectively unfindable until the cluster state fully converges

  • __share_group_state: Similar routing logic applies to share groups, risking state inconsistency.

...

Public Interfaces

  • Configuration
    • controller.unstable.allow.internal.coordinator.topicpartition.modificationincrease 
      • Type: boolean
      • Default: false
      • Description: When set to true, the controller allows partition expansion for internal topics. This is unsafe.
  • Error Codes
    • INVALID_REQUEST : Returned for specific internal topics within a CreatePartitions response when expansion is attempted without the unsafe config enabled.
  • classes

    • /kafka/core/src/main/scala/kafka/server/ControllerApis.scala → reject request

    • /kafka/core/src/main/scala/kafka/server/KafkaConfig.scala → Add configuration field for controller.unstable.allow.internal.coordinator.topicpartition.modificationincrease for ControllerApi.scala to access.

    • /kafka/raft/src/main/java/org/apache/kafka/raft/KRaftConfigs.java → Add controller.unstable.allow.internal.topiccoordinator.partition.modificationincrease.

Proposed Changes

We propose to enforce a validation check in the Controller API layer to reject CreatePartitions requests targeting internal topics,
while allowing other valid requests in the same batch to proceed.

...

We will introduce a new configuration to control this guardrail.

  • Config Name: controller.unstable.allow.internal.topiccoordinator.partition.modificationincrease

  • Definition: Defined in KRaftConfigs.java and propagated to KafkaConfig.scala.

  • Default: false

...

  1. The controller will iterate through the list of topics in the CreatePartitionsRequest.

  2. It will identify internal topics using Topic.isInternal(name).

  3. If controller.unstable.allow.internal.topiccoordinator.partition.modificationincrease is false:

    • Requests for internal topics will be immediately marked with INVALID_REQUEST error in the response.

    • Valid (non-internal) topics in the same request will be passed to controller.createPartitions for processing.

    • The final response will merge the results, returning per-topic success or failure codes.

...

  • Kafka 4.0 Context: This KIP targets KRaft-only clusters (Zookeeper is deprecated in 4.0).

  • Behavioral Change (Operational Breaking Change):

    • This KIP introduces a breaking change in behavior. Previously, expanding internal topics was technically allowed (though unsafe). After this change, such requests will fail with INVALID_REQUEST.

    • This is a deliberate design choice to prioritize correctness and cluster stability over backward compatibility of an unsafe operation.

    • Justification & Mitigation: This change is considered acceptable because Kafka documentation already explicitly advises against modifying internal topics. Furthermore, the impact is mitigated by the opt-in configuration (controller.unstable.allow.internal.coordinator.topicpartition.modificationincrease). Operators who strictly require this capability can explicitly enable it, ensuring that no critical workflows are permanently blocked.

  • Client Compatibility:

    • Existing AdminClient implementations (Java, Go, Python, etc.) do not need to be updated. They will simply receive a standard INVALID_REQUEST error code, which is already handled by standard error handling routines. No binary incompatibility is introduced.


...

  • No Data Migration Required: This change applies only to metadata operations. Existing data in internal topics is unaffected.

  • For Administrators:

    • If an administrator must expand partitions (e.g., during a critical manual recovery scenario), they must effectively "opt-in" to the risk by dynamically updating the controller configuration controller.allow.internal.topic.modification to true.Operators who explicitly need to perform this operation may opt in by setting the unsafe controller configuration to true and ensuring it is applied consistently across controllers. Depending on the deployment and configuration management approach, applying this setting may require a controller restart before it takes effect.

Test Plan

  • Unit Tests (ControllerApisTest):

    • Batch Handling: Verify that a request containing [valid-topic, __transaction_state] returns NONE (success) for the valid topic and INVALID_REQUEST for the internal topic.

    • Config Override: Verify that expansion succeeds when controller.unstable.allow.internal.topiccoordinator.partition.modificationincrease is true.

  • Integration Tests:

    • Use AdminClient against a KRaft cluster fixture to verify end-to-end blocking behavior.

...