...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

The Kafka Remote Log Manager currently uses three different thread pools to handle various types of tasks, but with inconsistent configuration naming and behavior, leading to several issues:

• Naming Inconsistency: The two newer thread pool configurations follow the pattern remote.log.manager.[task-type].thread.pool.size, but the original configuration remote.log.manager.thread.pool.size doesn't follow this pattern, despite now being specifically used for follower partition tasks.

• Inconsistent Dynamic Configuration Support: remote.log.manager.copier.thread.pool.size and remote.log.manager.expiration.thread.pool.size support dynamic configuration adjustments, while remote.log.manager.thread.pool.size does not, limiting system flexibility during runtime.

• Potential Confusion: The configuration name remote.log.manager.thread.pool.size suggests a general-purpose thread pool, but it now only handles follower partition tasks, which may mislead users about its actual purpose.

Although the original plan was to deprecate remote.log.manager.thread.pool.size

...

after the implementation of KIP-950, it was retained for follower tasks, and the intended refactoring was not fully carried out.

To resolve these issues, this KIP propose introducing a new dynamic configuration, remote.log.manager.follower.thread.pool.size, while marking the old configuration as deprecated. This change would enhance consistency and maintainability, providing users with a more intuitive and flexible configuration experience.

Public Interfaces

• Introduce a new broker config: remote.log.manager.follower.thread.pool.size

  Code Block
  Configuration: remote.log.manager.follower.thread.pool.size Description: Size of the thread pool used in scheduling follower tasks to read the highest-uploaded remote-offset for follower partitions. Type: Int Default: 2 Valid values: [1,...]

  
  

• Mark remote.log.manager.thread.pool.size as deprecated

Proposed Changes

The proposal introduces a new configuration for the Remote Log Manager: remote.log.manager.follower.thread.pool.size.

This configuration will support dynamic changes, allowing thread counts to be adjusted smoothly (with step increments from x/2 to x*2).

Compatibility, Deprecation, and Migration Plan

To ensure backward compatibility, the deprecated configuration remote.log.manager.thread.pool.size will still be functional. However, it should be removed in future major releases.

If both remote.log.manager.thread.pool.size and remote.log.manager.follower.thread.pool.size are set, the latter will take precedence.

Test Plan

The patch will include both unit and integration tests to ensure full coverage.

Rejected Alternatives

• Reusing the existing remote.log.manager.thread.pool.size and making it dynamic was considered. However, as outlined in the motivation section, the current config name is misleading and inconsistent with other thread pool size configurations.