...

This KIP borrows parts of KIP-170 discussion. If KIP-201 is resurrected, this changes shouldn't increase complexity of the KIP as it is just another field to map and same migration should apply.

Current workflow

At the moment there are 2 APIs for altering configs:

• (legacy) alterConfig
• incrementalAlterConfig

And alter config policy is treated somewhat different on each API.

The following workflows apply on each backend (ZK-based, and KRaft):

Workflows:

• Zookeeper-based (ZkAdminManager):
  • Legacy Alter Config
    • Received new configs (sum(c^i@n+1)) -- whole new config state
    • Filter nulls and pass values to new props (t@n+1)
    • Validate new props (t@n+1)
    • Apply policy to new configs (sum(c^i@n+1))
    • Store new props (t@n+1)
  • Incremental Alter Config
    • Collect changes (sum(c^i@n+1))
    • Fetch current props (t@n)
    • Apply each alter config op (c^i@n+1) and return new config (t@n+1)
    • Validate new props (t@n+1)
    • Apply policy to new configs (sum(c^i@n+1))
    • Store new props (t@n+1)

• KRaft (ConfigurationControlManager):
  • Legacy Alter Config
    • Where expected state (t@n+1) is all changes (sum(c^i@n+1))
    • Get current config (t@n)
    • For each change (c^i@n+1),
      • Get current config,
      • If diff value or broker change,
        • Add to records explicitly altered.
    • For each current config (c^i@n),
      • If not found in proposed (t@n+1),
      • Add to records implicitly deleted.
    • Validation
      • Get current config (t@n)
      • Prepare new config (t@n+1)
      • For each explicitly altered config
        • Apply change (c^i@n+1) to new config (t@n+1): including alter and deletes
        • Collect altered config on (sum(c^i@n+1))
      • For each implicit delete (only from legacy)
        • Apply delete to new config (t@n+1)
      • Validate new config (t@n+1)
      • Apply alter policy (sum(c^i@n+1))
      • Save new config (t@n+1)
  • Incremental Alter Config
    • For each change (c^i@n+1)
      • Get current config (c@n) and check current value
      • Prepare and add change request
    • Validation (change requests is explicitly alter configs, no implicit deletes)
      • Get current config (t@n)
      • Prepare new config (t@n+1)
      • For each explicitly altered config
        • Apply change (c^i@n+1) to new config (t@n+1): including alter and deletes
        • Collect altered config on (sum(c^i@n+1))
      • No implicit deletes on incremental
      • Validate new config (t@n+1)
      • Apply alter policy (sum(c^i@n+1))
      • Save new config (t@n+1)

Where:

• t@n: current set of configs
• t@n+1: next set of configs (i.e. once changes are applied)
• c^i: individual config key and value pair
• c^i@n+1: individual config update proposed
• sum(c^i@n+1): set of configs proposed

Some highlights:

• On Legacy Alter Config, the set of changes proposed (sum(c^i@n+1)) is the same as the new config (t@n+1) with the difference that null values are removed from the new config.
• On Incremental Alter Config, the set of changes proposed (sum(c^i@n+1)) is not the same as the new config (t@n+1), it only contains explicit changes to the config
• Implicit deletes are a set of configs created on legacy alter config, when no value is provided (not in t@n+1) but exists on the current config (t@n)
• Even though alter config policy receives the "requested" configurations, these have 2 different meanings depending on the API used to update configs.
  • When Legacy Alter Config, it means: requested changes (sum(c^i@n+1)) that is equal to new config state (t@n+1) including explicit deletes (sum(c^i@n+1 == null))
  • When Incremental Alter Config, it means: only requested changes (sum(c^i@n+1)) including explicit deletes (sum(c^i@n+1 == null)) but without any other config from current (!= t@n) or new status (!= t@n+1)
  • Plugin implementations do not know which one are they actually dealing with, and as incremental (new) API becomes broadly adopted, then configurations from the current status (t@n) not included in the request (sum(c^i@n+1)) are not considered.

Public Interfaces

1. Extend AlterConfigPolicy.RequestMetadata to include existing configurations:

...