DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
| Code Block | ||
|---|---|---|
| ||
{ "name": "TopologyDescriptionRequired", "type": "bool", "versions": "N+", "default": "false",
"about": "True if the broker's topology description plugin does not have an up-to-date topology description for this group. The client should send the topology description via UpdateStreamsGroupTopologyDescription." } |
The broker sets this field to to true when a topology description plugin is configured and plugin.requiresTopologyPush(requestContext, groupId, groupCreationTimeMs, topologyEpoch) returns true, where requestContext is the context of the heartbeat request.
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
package org.apache.kafka.coordinator.group.api.streams;
import org.apache.kafka.common.Configurable;
import org.apache.kafka.server.authorizer.AuthorizableRequestContext;
import java.util.concurrent.CompletableFuture;
/**
* A broker-side plugin that manages topology descriptions for streams groups.
*
* <p>Implementations receive topology descriptions pushed by Kafka Streams clients
* and can store, forward, or expose them however they see fit.
*
* <p>The broker calls {@link #requiresTopologyPush} on every heartbeat to determine
* whether the client should send its topology description. Implementations that need
* to consult an external service should kick off that work asynchronously on the
* first call and return {@code false} until the result is available.
*
* <p>Every method takes an {@link AuthorizableRequestContext} as its first argument.
*
* <p>Implementations must be thread-safe. {@link #setTopology} may be called
* concurrently by multiple group members observing
* {@code TopologyDescriptionRequired=true} in the same heartbeat cycle; concurrent
* calls with the same {@code (groupId, topologyEpoch)} pair carry identical data
* and must be treated as idempotent.
*/
public interface StreamsGroupTopologyDescriptionPlugin extends Configurable, AutoCloseable {
/**
* Returns whether the broker should request a topology push from the client.
*
* <p>Called on every successful heartbeat. Not called for members in
* {@code STALE_TOPOLOGY} status. If this method returns {@code true}, the broker
* sets {@code TopologyDescriptionRequired=true} in the heartbeat response.
*
* <p>This method should not throw. Failure modes should be handled internally
* and converted to a return value of {@code false}. The broker defensively
* catches any exception and treats it as {@code false}, logging at WARN; plugins
* should not rely on this backstop.
*
* <p>This method is on the heartbeat path and must return quickly. Implementations
* that need to consult an external service should return {@code false} until the
* result is available.
*
* <p>See the KIP's <em>Plugin Implementation Guidelines</em> section for how
* implementations should handle in-flight tracking, retries, and topology
* expiration.
*
* @param requestContext the context of the heartbeat request
* @param groupId the streams group ID
* @param groupCreationTimeMs the timestamp when the group was created. A value
* of {@code 0} means "unset". Plugins should treat {@code 0}
* as "incarnation indistinguishable" and should not cross-check
* stored epoch values against another stored description that
* also had {@code groupCreationTimeMs == 0}.
* @param topologyEpoch the topology epoch
* @return true if the broker should request a topology push from the client
*/
boolean requiresTopologyPush(AuthorizableRequestContext requestContext,
String groupId, long groupCreationTimeMs, int topologyEpoch);
/**
* Called when a client sends a topology description for a streams group.
* This method may be called concurrently by multiple members of the same group;
* all calls for the same (groupId, topologyEpoch) carry identical data.
*
* <p>The returned future completes when the topology has been persisted or
* forwarded. All failures must be signalled by completing the returned future
* exceptionally — implementations must not throw synchronously from this method.
* The broker handles the future's completion exception as follows:
* {@link org.apache.kafka.common.errors.InvalidRequestException} maps to
* {@code INVALID_REQUEST}; {@link org.apache.kafka.common.errors.TopologyDescriptionTooLargeException}
* maps to {@code TOPOLOGY_DESCRIPTION_TOO_LARGE}; any other exception maps to
* {@code TOPOLOGY_DESCRIPTION_UPDATE_FAILED} and is logged at WARN.
*
* @param requestContext the context of the UpdateStreamsGroupTopologyDescription request
* @param groupId the streams group ID
* @param groupCreationTimeMs the timestamp when the group was created
* @param topologyEpoch the topology epoch
* @param description the topology description
* @return a future that completes when the operation is done
*/
CompletableFuture<Void> setTopology(AuthorizableRequestContext requestContext,
String groupId, long groupCreationTimeMs, int topologyEpoch,
StreamsGroupTopologyDescription description);
/**
* Called when a group is explicitly deleted via DeleteGroups. Removes any topology
* description stored for this group.
*
* <p>The returned future completes when the deletion has been processed.
* If it completes exceptionally, the broker logs the error; the outcome does not
* affect the DeleteGroups response returned to the caller.
*
* @param requestContext the context of the DeleteGroups request
* @param groupId the streams group ID
* @return a future that completes when the operation is done
*/
CompletableFuture<Void> deleteTopology(AuthorizableRequestContext requestContext, String groupId);
/**
* Called to retrieve the stored topology description for a group. This is invoked
* by the broker when a client calls StreamsGroupDescribe with
* {@code IncludeTopologyDescription=true}.
*
* <p>Returns a future that resolves to the stored topology description for the
* given {@code (groupId, groupCreationTimeMs, topologyEpoch)} tuple, or to
* {@code null} if no topology is stored (e.g. no push has succeeded yet, or the
* stored description is for a different epoch). If the future completes
* exceptionally, the plugin signals a read error for this group.
*
* @param requestContext the context of the StreamsGroupDescribe request
* @param groupId the streams group ID
* @param groupCreationTimeMs the timestamp when the group was created
* @param topologyEpoch the topology epoch the caller is asking about
* @return a future resolving to the stored topology description, or null if none
*/
CompletableFuture<StreamsGroupTopologyDescription>
getTopology(AuthorizableRequestContext requestContext,
String groupId, long groupCreationTimeMs, int topologyEpoch);
} |
The plugin uses a a StreamsGroupTopologyDescription POJO that mirrors org.apache.kafka.streams.TopologyDescription but lives in the org.apache.kafka.coordinator.group.api.streams package; plugin implementations only need to depend on group-coordinator-api. The only difference is that there is no predecessor relation.
...
The plugin is instantiated at broker startup if
group.streams.topology.description.plugin.classis configured. A broker without a plugin does not advertiseUpdateStreamsGroupTopologyDescriptionin itsApiVersionsresponse and returnsUNSUPPORTED_VERSIONfor any such request. Because a plugin-less broker never setsTopologyDescriptionRequiredin heartbeat responses, in normal operation the RPC is only sent against a broker that has a plugin configured.After a successful
StreamsGroupHeartbeat, the broker callsrequiresTopologyPushon the plugin.STALE_TOPOLOGYmembers are skipped. If the plugin returnstrue, the broker setsTopologyDescriptionRequired=trueon the heartbeat response. Per the plugin contractrequiresTopologyPushshould not throw; the broker defensively catches any exception, logs at WARN, and treats the call asfalse. Beyond theSTALE_TOPOLOGYskip the broker does no other gating — deduplication, timeouts, and back-off are the plugin's responsibility (see Plugin Implementation Guidelines). ThegroupCreationTimeMsparameter identifies the specific incarnation of a group; a recreated group with the samegroupIdgets a fresh creation timestamp. It is persisted on the group record as a tagged field shared with KIP-1282. Groups created before KIP-1282 is implemented carry the sentinel valuegroupCreationTimeMs = 0.On- On
UpdateStreamsGroupTopologyDescription, the broker checks theREADACL on the group and that a plugin is configured. The broker does not enforce a size limit on the topology description — it is the plugin's responsibility to decide what size it is willing to store. The broker callssetTopologyon the plugin. On success, the response carriesNONE;InvalidRequestExceptionfrom the plugin maps toINVALID_REQUEST,TopologyDescriptionTooLargeExceptionmaps toTOPOLOGY_DESCRIPTION_TOO_LARGE, any other exception maps toTOPOLOGY_DESCRIPTION_UPDATE_FAILED, and all three are logged at WARN. - On
DeleteGroups, the broker callsdeleteTopologyon the plugin only after the group delete has returned no error for that group.deleteTopologyfailures are logged but do not affect the deletion response. For groups that expire naturally (all members leave),deleteTopologyis not called — the plugin expires the topology via a wall-clock TTL (see Plugin Implementation Guidelines). On
StreamsGroupDescribewithIncludeTopologyDescription=true, the broker callsgetTopologyon the plugin for each group after assembling the rest of the response, passing the group's current(groupCreationTimeMs, topologyEpoch)tuple. Calls across groups run in parallel. Authorization is unchanged — the existingDESCRIBEACL on the GROUP resource covers the topology description.DescribedGroup.ErrorCodeis never modified by topology-related outcomes; theTopologyDescriptionStatusfield carries the reason whenTopologyDescriptionis null (NOT_REQUESTED,NOT_STORED, orERROR; the last is also logged at WARN). When no plugin is configured, the broker returnsNOT_STORED(the broker does not distinguish "no plugin" from "plugin has nothing stored" on the wire).
...
Compatibility, Deprecation, and Migration Plan
This KIP bumps bumps StreamsGroupHeartbeatResponse, StreamsGroupDescribeRequest, and StreamsGroupDescribeResponse to the next available version of each RPC and adds the new fields (TopologyDescriptionRequired, IncludeTopologyDescription, TopologyDescription, TopologyDescriptionStatus) at that version. Pre-upgrade clients negotiate an older version and never see the new fields, so all three changes are wire-compatible.
...
During a rolling upgrade of brokers, some brokers may have the plugin configured and some may not. The The TopologyDescriptionRequired flag is only set by plugin-equipped coordinators; a client whose current coordinator lacks the plugin never sees the flag. If the coordinator migrates mid-push to a plugin-less broker, the client receives UNSUPPORTED_VERSION, clears the topologyDescriptionRequired flag, and does not retry. The flag is set again only if a future heartbeat response from a plugin-equipped coordinator includes TopologyDescriptionRequired=true.
...