...

This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: [One of "Under Discussion", "Accepted", "Rejected"]UNDER DISCUSS

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: here [Change the link from KAFKA-1 to your own ticket]

...

This makes it difficult to strengthen configuration constraints over time. For example, if the minimum allowed value for log.segment.bytes is increased (e.g., from 1 byte to 3 MB), brokers with legacy values in server.properties will fail to start after a binary upgrade due to a ConfigException. Because the broker cannot join the cluster, administrators cannot use dynamic APIs such as AlterConfigs to fix the configuration and must instead manually update configuration files across the fleet.

...

1. Broker static reporting: Brokers report their static (non-sensitive) configurations to the controller during registration. This enables the controller to perform pre-flight validation of static configs during metadata version upgrades. On the broker side, BrokerLifecycleManager is responsible for collecting and sending static configs. During registration, it iterates  over all non-internal config entries need to be validated.

2. Metadata-Version-Aware Validators: Configuration definitions are enhanced to support metadata-version-aware validators. The ConfigDef.define() method accepts a mvValidators parameter — Uses NavigableMap.floorEntry() to find the validator registered at the highest feature level ≤ the given featureLevel. This allows  constraints to be introduced incrementally across metadata versions.

   1. When an administrator runs kafka-storage format --release-version <version>, the tool now validates the broker's static configs from  server.properties against the MV constraints of the specified release version. If any config violates the constraints, the format command fails with a descriptive error before writing any data to disk.
   2. During AlterConfigs and IncrementalAlterConfigs operations, the Metadata-Version-Aware validators will be invoked to ensure that any proposed changes are compliant with the cluster's current metadata version. Any configuration that fails this runtime validation will be rejected with an INVALID_CONFIG error.
   3. When an administrator performs a metadata.
   4. During AlterConfigs and IncrementalAlterConfigs operations, the Metadata-Version-Aware validators will be invoked to ensure that any proposed changes are compliant with the cluster's current metadata version. Any configuration that fails this runtime validation will be rejected with an INVALID_CONFIG error.
   5. When an administrator performs a metadata.version upgrade or downgrade, the Controller will execute a comprehensive audit of all existing configurations—both static (via BrokerRegistrationRecord) and dynamic (via ConfigRecord). The transition will only proceed if all current settings satisfy the constraints of the target metadata version, ensuring a safe and atomic version migration.

   6. When an administrator runs kafka-storage format --release-version <version>, the tool now validates the broker's static configs from  server.properties against the MV constraints of the specified release version. If any config violates the constraints, the format command fails with a descriptive error before writing any data to disk.

Public Interfaces

ConfigDef

We are introducing a new mvValidators field to the ConfigDef class, enabling configurations to be validated against specific metadata.version thresholds. This allows the system to enforce version-specific constraints dynamically as the cluster evolves.

Example:

Suppose IBP_4_4_IV1 raises the minimum allowed value for log.segment.bytes from 1 byte to 3 MB. This constraint is registered as an MV-aware validator on the log.segment.bytes config key. Because log.segment.bytes now has an MV validator, it will be included in the static config payload that brokers send to the controller during registration.

1. Format time:  An administrator prepares a new broker with log.segment.bytes=1024 in server.properties and runs kafka-storage format --release-version 4.4. Before writing any data to disk, the tool validates the static configs against the constraints of IBP_4_4_IV1 and finds that log.segment.bytes=1024 violates the minimum of 3 MB. The command fails with a descriptive error, prompting the administrator to fix server.properties before retrying.
2. AlterConfigs at runtime: Once the cluster is running at IBP_4_4_IV1, an administrator attempts to set log.segment.bytes=1024 on a topic via AlterConfigs. The controller validates the proposed value against the current MV's constraints and immediately rejects the request with an INVALID_CONFIG error before any change is persisted.
3. Metadata version upgrade: A cluster is running at IBP_4_3_IV0, where brokers do not yet report static configs to the controller, leaving it with no visibility into each broker's server.properties. Broker node-3 has log.segment.bytes=2048 in its server.properties, but the controller is unaware of this. As the administrator performs a rolling binary upgrade, each restarted broker begins sending StaticConfigs in  BrokerRegistrationRequest version 6; since log.segment.bytes now has an MV validator registered, it is included in the payload and stored by the controller in memory even while the cluster is still at  IBP_4_3_IV0. When the administrator then requests an upgrade to IBP_4_4_IV1 — the version that simultaneously introduces static config reporting and the new log.segment.bytes minimum constraint — the  controller performs a pre-flight audit using the reported static configs and finds that node-3's log.segment.bytes=2048 violates the new minimum of 3 MB. The upgrade is rejected with a descriptive error identifying the broker and the offending value. Since the cluster is still at IBP_4_3_IV0 and fully operational, the administrator can issue an AlterConfigs request to set a cluster-wide dynamic override for log.segment.bytes to a compliant value, which takes precedence over the static value in server.properties without requiring any file editing or broker restart. The administrator then retries the upgrade, the pre-flight audit finds all effective configs compliant, and the upgrade proceeds successfully.

Public Interfaces

ConfigDef

We are introducing a new mvValidators field to the ConfigDef class, enabling configurations to be validated against specific metadata.version thresholds. This allows the system to enforce version-specific constraints dynamically as the cluster evolves.

    public static class ConfigKey {
        public final String name;
        public final Type type;
        public final String documentation;
        public final Object defaultValue;
        public final Validator validator;
        public final Importance importance;
        public final String group;
        public final int orderInGroup;
        public final Width width;
        public final String displayName;
        public final List<String> dependents;
        public final Recommender recommender;
        public final boolean internalConfig;
        public final String alternativeString;
        public final NavigableMap<Short, Validator> mvValidators; // new field
...
}

  /**
   * Define a new configuration with both a base validator and MV-specific validators.
   * @param name          the name of the config parameter
   * @param type          the type of the config
   * @param defaultValue  the default value to use if this config isn't present
   * @param validator     the base validator to use in checking the correctness of the config
   * @param mvValidators  a sorted map from metadata version feature level to the validator
   *                      that applies once that version is reached
   * @param importance    the importance of this config
   * @param documentation the documentation string for the config
   * @return This ConfigDef so you can chain calls
   */
  public ConfigDef define(String name, Type type, Object defaultValue,
                          Validator validator,
                          NavigableMap<Short, Validator> mvValidators,
                          Importance importance, String documentation)

...

Compatibility, Deprecation, and Migration Plan

• When admin upgrade or downgrade to specific metadata version, if there is any illegal config, it will return InvalidConfigError.

Test Plan

• Add  unit test and integration test to cover this new feature.

Rejected AlternativesIf there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way

1. Automatically adjust configuration values to meet new constraints: this approach was rejected because silently modifying configurations without administrator awareness is operationally dangerous and violates the principle of explicit configuration management.
2. Fail immediately with ConfigException on constraint violations: this approach was rejected because it creates operational deadlocks where brokers cannot start and administrators cannot use dynamic APIs to fix the configuration.