...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

...

Apache Kafka’s configuration validation is currently version-agnostic. Validation logic is defined in ConfigDef and executed during broker startup, which tightly couples the broker’s static codebase with the cluster’s operational configuration state.

This makes it difficult to strengthen configuration constraints over time. For example, if the minimum allowed value for log.segment.bytes is increased (e.g., from 1 byte to 3 MB), brokers with legacy values in server.properties will fail to start after a binary upgrade due to a ConfigException. Because the broker cannot join the cluster, administrators cannot use dynamic APIs such as AlterConfigs to fix the configuration and must instead manually update configuration files across the fleet.

To address the limitations of the current static, version-agnostic validation model, Kafka needs a mechanism to enforce configuration constraints

...

tied to

...

metadata.version

...

. As

...

Kafka evolves,

...

configuration values that were

...

previously valid

...

may become unsafe or semantically incorrect

...

in newer

...

versions. Without

...

version-

...

aware validation, two

...

issues arise:

1. Unsafe metadata version upgrades:

   Administrators may not be aware that their configuration changes were ignored, leading to unexpected cluster behavior.

   administrators may upgrade metadata.version without realizing that existing configurations violate new constraints.

2. No Metadata-version-aware enforcement: administrators can still use AlterConfigs or IncrementalAlterConfigs to set No runtime enforcement of version-specific constraints: Even after upgrading, administrators can use AlterConfigs or IncrementalAlterConfigs to set config values that violate constraints introduced by the current metadata version. There is no guardrail beyond the static ConfigDef validators, which are version-agnosticin newer metadata versions.

Additionally, the controller currently has no visibility into broker static configurations (those set in server.properties). Since these configs need to be changed dynamically, they must be validated before an upgrade is allowed — otherwise, a broker whose static config violates a new constraint would fail after the upgrade.

...

1. Broker static reporting: Brokers report their static (non-sensitive) configurations to the controller during registration. This enables the controller to perform pre-flight validation of static configs during metadata version upgrades.
2. Metadata-Version-Aware Validators::Configuration definitions enhance to support metadata-version-aware validators. The ConfigDef.define() method accepts a mvValidators parameter — a navigableMap from feature level (short) to ConfigDef.Validator instances.
   1. During AlterConfigs and IncrementalAlterConfigs operations, the Metadata-Version-Aware validators will be invoked to ensure that any proposed changes are compliant with the cluster's current metadata version. Any configuration that fails this runtime validation will be rejected with an INVALID_CONFIG error.
   2. When an administrator performs a metadata.version upgrade or downgrade, the Controller will execute a comprehensive audit of all existing configurations—both static (via BrokerRegistrationRecord) and dynamic (via ConfigRecord). The transition will only proceed if all current settings satisfy the constraints of the target metadata version, ensuring a safe and atomic version migration.

...