DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
Since the full topic is read anyway, we should have a method to translate the committed offsets of multiple consumer groups at the same time.
Public Interfaces
A new translateOffsetsnew translateOffsets() method in RemoteClusterUtils, that takes a set of consumer group Idsregex pattern to specify the consumer groups:
| Code Block | ||
|---|---|---|
| ||
/**
* Translates remote consumer groups' offsets into corresponding local offsets. Topics are automatically
* renamed according to the configured {@link ReplicationPolicy}.
* @param properties Map of properties to instantiate a {@link MirrorClient}
* @param remoteClusterAlias The alias of the remote cluster
* @param consumerGroupIdsconsumerGroupPattern The set ofregex pattern specifying the consumer group Ids
groups to translate offsets for
* @param timeout The maximum time to block when consuming from the checkpoints topic
*/
public static Map<String, Map<TopicPartition, OffsetAndMetadata>> translateOffsets(Map<String, Object> properties, String remoteClusterAlias, Set<String>Pattern consumerGroupIdsconsumerGroupPattern, Duration timeout) {
} |
The matching method in MirrorClient:
| Code Block | ||
|---|---|---|
| ||
/** * Translates remote consumer groups' offsets into corresponding local offsets. Topics are automatically * renamed according to the ReplicationPolicy. * @param consumerGroupIdsconsumerGroupPattern The regex setpattern specifying ofthe consumer group Ids groups to translate offsets for * @param remoteClusterAlias The alias of remote cluster * @param timeout The maximum time to block when consuming from the checkpoints topic */ public Map<String, Map<TopicPartition, OffsetAndMetadata>> remoteConsumerOffsets(Set<String>Pattern consumerGroupIdsconsumerGroupPattern, String remoteClusterAlias, Duration timeout) { } |
For both methods, in case there are no offsets for a group, the Map will still contain an entry with the group Id as the key but the value will be an empty Map.
Proposed Changes
The existing MirrorClient.remoteConsumerOffsets() will invoke the new method with a Set containing just a single consumer group Id.
...
- Deprecate the existing RemoteClusterUtils.translateOffsets() and MirrorClient.remoteConsumerOffsets() methods: Internally these will use the same logic as the new methods, so we can keep them.
- Specify the desired consumer groups via a collection: This limits the ability of translating all consumer group offsets.