Versions Compared

Old Version 57

changes.mady.by.user Satish Duggana

Saved on

compared with

New Version 58

changes.mady.by.user Satish Duggana

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Compacted topics will not have remote storage support. 

Configs

System-Wide

remote.log.storage.enable = false (to support backward compatibility)

remote.log.storage.manager.class.name name =  org.apache.kafka.rsm.hdfs.HDFSRemoteStorageManager

RemoteStorageManager

(These configs are dependent on remote storage manager implementation)

remote.log.storage.*

Thread pools

remote.log.manager.thread.pool.size
Remote log thread pool size, which is used in scheduling tasks to copy segments, fetch remote log indexes and clean up remote log segments.

remote.log.manager.task.interval.ms
Interval at which remote log manager runs the scheduled tasks like copy segments, fetch remote log indexes and clean up remote log segments.

remote.log.reader.threads
Remote log reader thread pool size

remote.log.reader.max.pending.tasks
Maximum remote log reader thread pool task queue size. If the task queue is full, broker will stop reading remote log segments.

Per Topic Configuration

remote.log.retention.minutes

remote.log.retention.bytes

Remote Storage Manager:

         `RemoteStorageManager` is an interface to provide the lifecycle of remote log segments. We will provide a simple implementation of RSM to get a better understanding of the APIs. HDFS and S3 implementation are planned to be hosted in external repos and these will not be part of Apache Kafka repo. This is inline with the approach taken for Kafka connectors.

...