Current state: "Under Discussion"
Discussion thread:
JIRA:
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
The Kafka Remote Log Manager currently uses three different thread pools to handle various types of tasks, but with inconsistent configuration naming and behavior, leading to several issues:
Naming Inconsistency: The two newer thread pool configurations follow the pattern remote.log.manager.[task-type].thread.pool.size, but the original configuration remote.log.manager.thread.pool.size doesn't follow this pattern, despite now being specifically used for follower partition tasks.
Inconsistent Dynamic Configuration Support: remote.log.manager.copier.thread.pool.size and remote.log.manager.expiration.thread.pool.size support dynamic configuration adjustments, while remote.log.manager.thread.pool.size does not, limiting system flexibility during runtime.
Potential Confusion: The configuration name remote.log.manager.thread.pool.size suggests a general-purpose thread pool, but it now only handles follower partition tasks, which may mislead users about its actual purpose.
Although the original plan was to deprecate remote.log.manager.thread.pool.size
after the implementation of KIP-950, it was retained for follower tasks, and the intended refactoring was not fully carried out.
To resolve these issues, this KIP propose introducing a new dynamic configuration, remote.log.manager.follower.thread.pool.size, while marking the old configuration as deprecated. This change would enhance consistency and maintainability, providing users with a more intuitive and flexible configuration experience.
Introduce a new broker config: remote.log.manager.follower.thread.pool.size
Configuration: remote.log.manager.follower.thread.pool.size Description: Size of the thread pool used in scheduling follower tasks to read the highest-uploaded remote-offset for follower partitions. Type: Int Default: 2 Valid values: [1,...] |
Mark remote.log.manager.thread.pool.size as deprecated
The proposal introduces a new configuration for the Remote Log Manager: remote.log.manager.follower.thread.pool.size.
This configuration will support dynamic changes, allowing thread counts to be adjusted smoothly (with step increments from x/2 to x*2).
To ensure backward compatibility, the deprecated configuration remote.log.manager.thread.pool.size will still be functional. However, it should be removed in future major releases.
If both remote.log.manager.thread.pool.size and remote.log.manager.follower.thread.pool.size are set, the latter will take precedence.
The patch will include both unit and integration tests to ensure full coverage.
remote.log.manager.thread.pool.size and making it dynamic was considered. However, as outlined in the motivation section, the current config name is misleading and inconsistent with other thread pool size configurations.