DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
This section describes new configurations introduced by Cluster Mirroring.
...
Broker
Set via CreateMirror or IncrementalAlterConfigsbroker config. Stored in cluster metadata recordsserver.properties or dynamic broker config.
Key | Description | Default |
|---|
bootstrap.servers
A list of host/port pairs to use for establishing the initial connection to the source cluster.
mirror.topic.num.partitions | Number of partitions for __mirror_state internal topic. | 50 |
mirror.topic. |
replication.factor | Replication factor for __mirror_state internal topic. | 3 |
mirror.num.replica.fetchers | Number of fetcher threads per mirrored source broker, | 1 |
mirror.metadata.refresh.interval.ms | The interval in milliseconds at which the coordinator refreshes metadata from source clusters. This controls how frequently the coordinator polls source clusters to detect new topics and metadata changes. | 30000 |
mirror.replication.throttled.rate | A long representing the upper bound (bytes/sec) on replication traffic for mirrored follower node enumerated in the property “mirror.replication.throttled.replicas” (for each topic). This property can be only set dynamically. It is suggested that the limit be kept above 1MB/s for accurate behaviour. | MAX_LONG |
request.timeout.ms | Maximum amount of time in milliseconds the client will wait for the response of a request. | 30000 |
socket.* | Socket connection configurations. | |
replica.* | Fetcher threads configurations. |
Mirror
Set via CreateMirror or IncrementalAlterConfigs. Stored in cluster metadata records.
Key | Description | Default | |||||
|---|---|---|---|---|---|---|---|
bootstrap.servers | A list of host/port pairs to use for establishing the initial connection to the source cluster. | ||||||
mirror.topic.properties | excludeA comma-separated list of topic config property names to exclude from synchronization. Properties in this list will not be replicated from the source cluster. The mirror.name property is always excluded regardless of this setting. | follower.replication.throttled.replicas, leader.replication.throttled.replicas, message.timestamp.difference.max.ms, log.message.timestamp.before.max.ms, log.message.timestamp.after.max.ms, message.timestamp.type, unclean.leader.election.enable, min.insync.replicas, mirror.name | mirror.topics.include | A comma-separated list of regex patterns for topic names to include in mirroring. Topics on the source cluster whose names match at least one of the patterns will be automatically discovered and mirrored.
| mirror.topics.exclude | A comma-separated list of regex patterns for topic config property names to exclude from synchronization. Properties in this list will not be replicated from the source cluster. The mirror.name property is always excluded regardless of this setting. | follower.replication.throttled.replicas, leader.replication.throttled.replicas, message.timestamp.difference.max.ms, log.message.timestamp.before.max.ms, log.message.timestamp.after.max.ms, message.timestamp.type, unclean.leader.election.enable, min.insync.replicas, mirror.name |
mirror.topics.include | A comma-separated list of regex patterns for topic names to include in mirroring. Topics on the source cluster whose names match at least one of the patterns will be automatically discovered and mirrored.
| ||||||
mirror.topics.exclude | A comma-separated list of regex patterns for topic names to exclude from mirroring. Topics matching the exclude pattern are not mirroring. Topics matching the exclude pattern are not mirrored even if they match mirror.topics.include. Internal topics are always excluded. Exclude always wins over include. By default internal topics (starting with __ ) are excluded. | ||||||
mirror.groups.include | A comma-separated list of regex patterns for group IDs to include in offset synchronization. Only groups whose IDs match at least one of the patterns will have their offsets replicated from the source cluster. | ||||||
| mirror.groups.exclude | A comma-separated list of regex patterns for group IDs to exclude from offset synchronization. Groups matching the exclude pattern are not replicated even if they match mirror.groups.include. | ||||||
mirror.acl.include | A comma-separated list of ACL include rules. Each rule uses semicolon-separated fields: resourceType;resourceName;operation;permissionType;principal. Use '*' as wildcard for any field. The resourceName field supports regex patterns. Trailing wildcard fields can be omitted. See AclRule javadoc for examples. Examples: TOPIC;orders.* (all ACLs for topics matching orders.*) *;*;*;*;User:alice (all ACLs for principal User:alice) *;*;*;*;User:app-.* (all ACLs for principals matching User:app-.*) TOPIC;*;READ;ALLOW (all topic READ/ALLOW ACLs) GROUP;consumer-.*;READ;ALLOW;User:bob (READ/ALLOW ACLs on groups matching consumer-.* for User:bob) TOPIC;orders.*,*;*;*;*;User:alice (sync all topic ACLs for orders.* topics and all ACLs for User:alice) | * | |||||
security.protocol | Protocol for source cluster communication (PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL). | ||||||
sasl.* | SASL configuration properties. | ||||||
ssl.* | SSL configuration properties. |
...
Key | Description | Default |
|---|---|---|
mirror.name | Identifies the mirror that manages this topic. Topics with this configuration set are read-only and can only be modified through mirror management APIs. This property is filtered out from DescribeConfigs responses to avoid exposing internal state to users. | |
mirror.replication.throttled.replicas | A list of replicas for which log replication should be throttled on the mirror follower node. The list should describe a set of replicas in the form [PartitionId]:[BrokerId],[PartitionId]:[BrokerId]:... or alternatively the wildcard '*' can be used to throttle all replicas for this topic." | |
mirror.support.unclean.leader.election | When enabled, LME log truncation waits for all replicas (not just ISR members) to join the ISR and complete the truncation. | false |
Broker
Set via broker config. Stored in server.properties or dynamic broker config.
Key
Description
Default
mirror.topic.num.partitions
Number of partitions for __mirror_state internal topic.
50
mirror.topic.replication.factor
Replication factor for __mirror_state internal topic.
3
mirror.num.replica.fetchers
Number of fetcher threads per mirrored source broker,
1
mirror.metadata.refresh.interval.ms
The interval in milliseconds at which the coordinator refreshes metadata from source clusters. This controls how frequently the coordinator polls source clusters to detect new topics and metadata changes.
30000
mirror.replication.throttled.rate
A long representing the upper bound (bytes/sec) on replication traffic for mirrored follower node enumerated in the property “mirror.replication.throttled.replicas” (for each topic). This property can be only set dynamically. It is suggested that the limit be kept above 1MB/s for accurate behaviour.
request.timeout.ms
Maximum amount of time in milliseconds the client will wait for the response of a request.
30000
socket.*
Socket connection configurations.
replica.*
Metrics
A core set of metrics will be provided with the initial implementation.
...