You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 20 Next »

Status

Current state: Under Discussion

Discussion thread: https://lists.apache.org/thread/l8ko353v3nn1blgymsty895x6c98oxlx

JIRA: KAFKA-17747 - Getting issue details... STATUS

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

KIP-848 introduces a next generation of consumer rebalance protocol which supports rack-aware partition assignment. In the first implement, the group coordinator computes subscribed topic metadata which contains topic UUID, name, number of partition, and rack set of each partition. When the metadata is expired, the group coordinator computes new subscribed topic metadata and compare with current one. If result is difference, the group coordinator bumps group epoch and calculates new target assignment which means it triggers a rebalance. However, the rack set of each partition takes too much space. In KAFKA-17578, a real case of memory usage shows that in a group with 500 members and 2K topic partitions,  partition racks account for 79% of whole ConsumerGroup object. This KIP will get rid of number of partition and rack set of each partition. It uses a subscribed topic hash to replace it. Each topic has a hash value. There are two server side conditions can trigger a new rebalance, so the topic hash needs to reflect them:

  • A topic with a new partition.
  • A topic partition has rack change. Each topic partition has multiple replicas. Each replica is stored on a broker. The rack value is from the broker.rack config. It's a read only config and can only be changed when broker restarts. If a broker stops, a related replica will be removed from topic partitions. If a broker starts, it will be added to topic partitions.

Following table compares cpu / memory usage of different strategies:


CPUMemory

1. Subscription topic metadata with topic UUID, name, number of partition, and rack set of each partition.

LowHigh
2. Subscription topic metadata with topic UUID, name, and hash.MidMid
3. A single hash to represent all subscribed topic in a group.HighLow

We choose the second way, but not the third way, because a single hash to represent all subscribed topic in a group wastes too much CPU resource to recalculate the hash when the metadata image is expired. If any subscribed topic change, the third way needs to aggregate all data an recalculate the hash. The second way only needs to recalculate expired topic hash.

Public Interfaces

ConsumerGroupPartitionMetadataValue

Add a new tagged field TopicHash. 

ConsumerGroupPartitionMetadataValue
{
  "type": "data",
  "name": "ConsumerGroupPartitionMetadataValue",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "Topics", "versions": "0+", "type": "[]TopicMetadata",
      "about": "The list of topic metadata.", "fields": [
      { "name": "TopicId", "versions": "0+", "type": "uuid",
        "about": "The topic id." },
      { "name": "TopicName", "versions": "0+", "type": "string",
        "about": "The topic name." },
      { "name": "NumPartitions", "versions": "0+", "type": "int32",
        "about": "Deprecated: this field is not used after 4.0. The number of partitions of the topic." }, <-- deprecated field
      { "name": "PartitionMetadata", "versions": "0+", "type": "[]PartitionMetadata",
        "about": "Deprecated: this field is not used after 4.0. Partitions mapped to a set of racks. If the rack information is unavailable for all the partitions, an empty list is stored", "fields": [
          { "name": "Partition", "versions": "0+", "type": "int32",
            "about": "The partition number." },
          { "name": "Racks", "versions": "0+", "type": "[]string",
            "about": "The set of racks that the partition is mapped to." }
      ]}, <-- deprecated field
      { "name": "TopicHash", "versions": "0+", "type": "int64",
        "default": 0, "taggedVersions": "0+", "tag": 0,
        "about": "The hash value of the topic id, name, number of partitions, and partition racks." } <-- new field
    ]}
  ]
}

ShareGroupPartitionMetadataValue

Add a new tagged field TopicHash.

ShareGroupPartitionMetadataValue
{
  "type": "data",
  "name": "ShareGroupPartitionMetadataValue",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "Topics", "versions": "0+", "type": "[]TopicMetadata",
      "about": "The list of topic metadata.", "fields": [
      { "name": "TopicId", "versions": "0+", "type": "uuid",
        "about": "The topic id." },
      { "name": "TopicName", "versions": "0+", "type": "string",
        "about": "The topic name." },
      { "name": "NumPartitions", "versions": "0+", "type": "int32",
        "about": "Deprecated: this field is not used after 4.0. The number of partitions of the topic." }, <-- deprecated field
      { "name": "PartitionMetadata", "versions": "0+", "type": "[]PartitionMetadata",
        "about": "Deprecated: this field is not used after 4.0. Partitions mapped to a set of racks. If the rack information is unavailable for all the partitions, an empty list is stored", "fields": [
        { "name": "Partition", "versions": "0+", "type": "int32",
          "about": "The partition number." },
        { "name": "Racks", "versions": "0+", "type": "[]string",
          "about": "The set of racks that the partition is mapped to." }
      ]}, <-- deprecated field
      { "name": "TopicHash", "versions": "0+", "type": "int64",
        "default": 0, "taggedVersions": "0+", "tag": 0,
        "about": "The hash value of the topic id, name, number of partitions, and partition racks." } <-- new field
    ]}
  ]
}

Proposed Changes

Topic Hash Map Cache in Coordinator

Different groups may subscribe to same topics. With topic hash map cache in the coordinator, it can avoid recalculation of same topic hash for different groups. The topic hash value should represent topic id, name, number of partitions, and partition racks.

Initial a topic hash in the cache

The coordinator only cares topics which are subscribed by groups. When a coordinator starts, it replays all records in __consumer_offsets. When replaying ConsumerGroupMemberMetadtaValue / ShareGroupMemberMetadataValue, the group can get subscribed topic names from previous state. In the next group heartbeat, the group can rely on subscribed topic names to compute topic hash and keep it in the cache, so other groups don't need to recompute it.

Remove a topic hash in the cache

If a topic is not subscribed by any group, the coordinator doesn't need to maintain the hash value. When replaying ConsumerGroupMemberMetadtaValue / ShareGroupMemberMetadataValue records, the coordinator adds a new topic to groupsByTopics if the topic is absent. If a group unsubscribes a topic, the group is removed from groupsByTopics. If there is no group under a topic, the topic will be removed from groupsByTopics. The group coordinator can leverage same mechanism to remove the topic hash.

Renew a topic hash in the cache

When there is a new metadata image, it contains changed topics and deleted topics in metadata delta. For both cases, the coordinator remove topic hash from the cache. The value will be recomputed when in the next consumer group heartbeat. The lazy evaluation is more efficient. If there is no further group heartbeat, the new topic hash value is useless, so we can just compute it in group heartbeat.


public class GroupMetadataManager {
    // ...

    /**
     * The topic hash value by topic name.
     */
    private final Map<String, Long> topicHashCache; 

    /**
     * Unsubscribes a group from a topic.
     *
     * @param groupId   The group id.
     * @param topicName The topic name.
     */
    private void unsubscribeGroupFromTopic(
        String groupId,
        String topicName
    ) {
        groupsByTopics.computeIfPresent(topicName, (__, groupIds) -> {
            groupIds.remove(groupId);
            if (groups.isEmpty()) {
                topicHashCache.remove(topicName);
                return null;
            }
            return groupIds;
        });
    }

    /**
     * A new metadata image is available.
     *
     * @param newImage  The new metadata image.
     * @param delta     The delta image.
     */
     public void onNewMetadataImage(MetadataImage newImage, MetadataDelta delta) {
        metadataImage = newImage;

        // Initialize the last offset if it was not yet.
        if (lastMetadataImageWithNewTopics == -1L) {
            lastMetadataImageWithNewTopics = metadataImage.provenance().lastContainedOffset();
        }

        TopicsDelta topicsDelta = delta.topicsDelta();
        if (topicsDelta == null) return;

        // Updated the last offset of the image with newly created topics. This is used to
        // trigger a refresh of all the regular expressions when topics are created. Note
        // that we don't trigger a refresh when topics are deleted. Those are removed from
        // the subscription metadata (and the assignment) via the above mechanism. The
        // resolved regular expressions are cleaned up on the next refresh.
        if (!topicsDelta.createdTopicIds().isEmpty()) {
            lastMetadataImageWithNewTopics = metadataImage.provenance().lastContainedOffset();
        }

        // Notify all the groups subscribed to the created, updated or
        // deleted topics.
        Set<String> allGroupIds = new HashSet<>();
        topicsDelta.changedTopics().forEach((topicId, topicDelta) -> {
            String topicName = topicDelta.name();
            // trigger recalculate topic hash in next consumer group heartbeat
            topicHashCache.remove(topicName);
            allGroupIds.addAll(groupsSubscribedToTopic(topicName));
        });
        topicsDelta.deletedTopicIds().forEach(topicId -> {
            TopicImage topicImage = delta.image().topics().getTopic(topicId);
            String topicName = topicImage.name();
            topicHashCache.remove(topicName);
            allGroupIds.addAll(groupsSubscribedToTopic(topicName));
        });
        allGroupIds.forEach(groupId -> {
            Group group = groups.get(groupId);
            if (group != null && (group.type() == CONSUMER || group.type() == SHARE)) {
                ((ModernGroup<?>) group).requestMetadataRefresh();
            }
        });
    }
}

Subscribed Topic Hash Map in Group

To compute a new subscribed topic hash map, a group can leverage cache in the coordinator. If a topic hash is calculated by another group, we can just reuse it from cache.

Topic Hash Function

A topic hash represent topic id, name, number of partitions, and partition racks. When restarts the coordinator, the topic hash will be recomputed and keep in topic hash map. To avoid useless rebalance, the hash function should return same value for same data, even if it runs on different JDKs or partition racks have different order. For different JDKs, the KIP will use Murmur3 to compute the hash value. The function is already in org.apache.kafka.streams.state.internals. We will move it to org.apache.kafka.common.hash for common usage. For partition racks with different order, we will compute hash for each value in it and sum as a result.

public abstract class ModernGroup<T extends ModernGroupMember> implements Group {
    // ...

    /**
     * Computes the subscription metadata based on the current subscription info.
     * 
     * @param subscribedTopicNames Map of topic names to the number of subscribers. 
     * @param metadataImage        The current metadata for all available topics.
     * @param topicHashCache       The cache of topic hashes.
     * @return An immutable map of subscription topic hash for each topic that the consumer group is subscribed to.
     */
    public Map<String, TopicMetadata> computeSubscriptionMetadata(
        Map<String, SubscriptionCount> subscribedTopicNames,
        MetadataImage metadataImage,
        Map<String, Long> topicHashCache
    ) {
        // Create the topic metadata for each subscribed topic.
        Map<String, TopicMetadata> newSubscriptionMetadata = new HashMap<>(subscribedTopicNames.size());
        TopicsImage topicsImage = metadataImage.topics();
        subscribedTopicNames.forEach((topicName, count) -> {
            TopicImage topicImage = topicsImage.getTopic(topicName);
            if (topicImage != null) {
                newSubscriptionMetadata.put(topicName, new TopicMetadata(
                    topicImage.id(),
                    topicImage.name(),
                    topicHashCache.computeIfAbsent(
                        topicName,
                        key -> computeTopicHash(topicImage, metadataImage.cluster())
                    )
                ));
            }
        });

        return Collections.unmodifiableMap(newSubscriptionMetadata);
    }

    /**
     * Computes the hash of the topic id, name, number of partitions, and partition racks by Murmur3.
     *
     * @param topicImage   The topic image.
     * @param clusterImage The cluster image.
     */
    public static long computeTopicHash(TopicImage topicImage, ClusterImage clusterImage) {
        long result = topicImage.id().hashCode();
        result = 63 * result + Murmur3.hash64(topicImage.name().getBytes(StandardCharsets.UTF_8));
        result = 63 * result + Murmur3.hash64(topicImage.partitions().size());
        result = 63 * result + topicImage.partitions().entrySet().stream().mapToLong(
            entry -> {
                PartitionRegistration partitionRegistration = entry.getValue();
                long partitionHash = Murmur3.hash64(entry.getKey());
                Set<String> racks = Arrays.stream(partitionRegistration.replicas)
                    .mapToObj(clusterImage::broker)
                    .map(BrokerRegistration::rack)
                    .filter(Optional::isPresent)
                    .map(Optional::get)
                    .collect(Collectors.toSet());
                return 63 * partitionHash + racks.stream().mapToLong(
                    rack -> Murmur3.hash64(rack.getBytes(StandardCharsets.UTF_8))
                ).reduce(0L, Long::sum);
            }
        ).reduce(0L, Long::sum);
        return result;
    }
}

SubscribedTopicDescriberImpl

The topic metadata doesn't have number of partitions and rack data, so the subscribedTopicDescriberImpl can't rely on it to return numPartitions and racksForPartition. This KIP adds MetadataImage to SubscribedTopicDescriberImpl constructor, so it can use Map<Uuid, TopicMetadata> to check which topic is subscribed and use MetadataImage to return numPartitions and racksForPartition.

Compatibility, Deprecation, and Migration Plan

ConsumerGroupPartitionMetadataValue / ShareGroupPartitionMetadataValue

There is new field in ConsumerGroupPartitionMetadataValue and ShareGroupPartitionMetadataValue. For compatibility, adding it as tagged field, so old brokers can deserialize it.

Test Plan

  • Unit test for the topic hash function. The hash function should ignore partition racks order and give a same value if the set is no difference.
  • Unit test for GroupMetadataManager. Test only topic id, name, number of partition, and partition racks change can make a rebalance. Other data change should keep same assignment.
  • Integration test between coordinator and consumer. Use admin client to update topic partition and check consumer group get a new assignment. Restart a broker to change rack and check consumer group get a new assignment. Use a consumer group to subscribe regex pattern and use admin client to add a new pattern matched topic, check the consumer group get a new assignment.

Rejected Alternatives

Single Hash

Using a single hash to represent subscribed topic metadata can detect when to trigger a rebalance. This approach also reduce memory and disk usage. However, it wastes lot of resources to recalculate same topic hash. For example, if 10 groups subscribe to a same topic, the coordinator need to recalculate it 10 times.

Check Topic Delta to Trigger a Rebalance

Check the changed part like TopicDelta#newPartitons when receiving a new metadata image. If there is change data, set a value in the group and trigger a rebalance in the next group heartbeat. This approach can reduce hash calculation and storage. However, the group coordinator is not always online. If the topic change is not happened with online coordinator, the change will be ignored and the coordinator can't trigger a rebalance.

Add Epoch to TopicImage

A new metadata image is computed by controller. An epoch can represent the version of TopicImage. If there is number of partitions or partition racks change, the controller bumps the epoch. When coordinator receives a new metadata image and there is a new topic epoch, it triggers a rebalance for a group. The group also can store topic epoch map in records, so it can know the difference if the coordinator restarts. This approach save cpu and storage resources and avoid the downside of "Check Topic Delta to Trigger a Rebalance". However, this approach mixes the group coordinator logic within the controller.


  • No labels