Page History

...

System-Wide	remote.log.storage.enable - Whether to enable remote log storage or not. Valid values are `true` or `false` and the default value is false. This property gives backward compatibility. remote.log.storage.manager.class.name - This is mandatory if the remote.log.storage.enable is set as true. remote.log.metadata.manager.class.name(optional) - This is an optional property. If this is not configured, Kafka uses an inbuilt metadata manager backed by an internal topic.
RemoteStorageManager	(These configs are dependent on remote storage manager implementation) remote.log.storage.*
RemoteLogMetadataManager	(These configs are dependent on remote log metadata manager implementation) remote.log.metadata.*
Thread pools	remote.log.manager.thread.pool.size Remote log thread pool size, which is used in scheduling tasks to copy segments, fetch remote log indexes and clean up remote log segments. remote.log.manager.task.interval.ms The interval at which remote log manager runs the scheduled tasks like copy segments, fetch remote log indexes and clean up remote log segments. remote.log.reader.threads Remote log reader thread pool size remote.log.reader.max.pending.tasks Maximum remote log reader thread pool task queue size. If the task queue is full, broker will stop reading remote log segments.
Per Topic Configuration	remote.log.retention.minutesms remote.log.retention.bytes

...

The following new metrics will be added:

mbean	description
kafka.log.remote:type=RemoteLogReaderMetrics, name=RequestsPerSec, topic=([-.w]+)	Number of remote storage read requests per second.
kafka.log.remote:type=RemoteLogReaderMetrics, name=BytesPerSec, topic=([-.w]+)	Number of bytes read from remote storage per second.
kafka.log.remote:type=RemoteLogReaderMetrics, name=ErrorsPerSec	Number of remote storage read errors per second.
kafka.log.remote:type=RemoteStorageThreadPool, name=RemoteLogReaderTaskQueueSize	Number of remote storage read tasks pending for execution.
kafka.log.remote:type=RemoteStorageThreadPool, name=RemoteLogReaderAvgIdlePercent	Average idle percent of the remote storage reader thread pool.
kafka.log.remote:type=RemoteLogManagerMetrics, name=RemoteLogManagerTasksAvgIdlePercent	Average idle percent of RemoteLogManager thread pool.
kafka.log.remote:type=RemoteLogManagerMetrics, name=CopyToRemoteStorageBytesPerSec	Number of bytes copied to remote storage per second.
kafka.log.remote:type=RemoteLogManagerMetrics, name=CopyToRemoteStorageErrorsPerSec, topic=([-.w]+)	Number of remote storage write errors per second.
kafka.log.remote:type=RemoteLogManagerMetrics, name=CopyToRemoteStorageBacklogBytes	The total number of bytes of the segments that are pending to be copied to remote storage.

Performance Test Results

We have tested the performance of the initial implementation of this proposal.

...