Versions Compared

Old Version 1

changes.mady.by.user Jhen-Yung Hsu

Saved on

compared with

New Version 2

changes.mady.by.user Jhen-Yung Hsu

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Discussion thread: Under Discussion

JIRA: here

Motivation

The configuration can be applied at the topic, broker, or cluster level. However, our primary concern is the behavior of topics, or ensuring that topics operate consistently across the cluster, rather than expecting the same topic to behave differently on different brokers. For example, if there are two brokers with different settings for min.insync.replicas, moving a leader to a different broker would change its behavior with respect to the min.insync.replicas semantics.

This KIP aims to enforce that the proposed configurations have the same cluster-level settings and disallow them from being set at the broker level, ensuring consistent behavior across the cluster.

...

  • On the cluster's first startup, initialize proposed configurations at the cluster level with their static default values.
  • Disallow setting these configurations at the broker level via the incrementalAlterConfigs API or the deprecated AlterConfigs API. The new error, DISALLOWED_BROKER_LEVEL_CONFIG_ERROR, will be returned if such requests are made.
  • Disallow removing these configurations at the cluster level via the incrementalAlterConfigs API or the deprecated AlterConfigs API. The new error, DISALLOWED_CLUSTER_LEVEL_CONFIG_REMOVAL_ERROR, will be returned if such requests are made.

  • When upgrading to a version that includes this KIP, any proposed configurations at the broker level will be removed. For cluster level, if they are not set, they will be initialized to their default values. The default values are the static configs.

New Errors

  • DISALLOWED_BROKER_LEVEL_CONFIG_ERROR
  • DISALLOWED_CLUSTER_LEVEL_CONFIG_REMOVAL_ERROR

...

Compatibility, Deprecation, and Migration Plan

  • For the incrementalAlterConfigs and AlterConfigs APIs, any disallowed requests will be rejected and immediately throw an error. This may break user applications.

  • On upgrade, any broker-level settings for the proposed configurations will be removed. This may break users who set these configs at the broker level.

...

The typical suite of unit/integration tests will be added.

Rejected Alternatives

1. Disallow alter API reqeusts with warnings and ignore setting

Instead of immediately rejecting disallowed requests with an error, only log a warning and ignore the setting. This would avoid breaking user applications until the next major release.

Reason for Rejection: Users may easily overlook warnings and not realize their configurations have not taken effect, which could lead to even more severe operational issues later.

2. Each proposed config with its own logic

Instead of introducing a new enum structure, handle each configuration individually with its own logic (similar to how min.insync.replicas is currently implemented, see https://github.com/apache/kafka/pull/17952), 

Reason for Rejection: This approach is harder to extend and more error-prone. Having a centralized structure provides a cleaner and more extensible solution.