...

• Implement requiresTopologyPush as a non-blocking, efficient call, since it is invoked frequently.
• Expire and clean up stored topology descriptions for inactive groups and superseded topology epochs internally. Incoming requiresTopologyPush calls can be used as an implicit keep-alive.
• Decide and enforce a maximum stored description size. Reject pushes that exceed it by completing the setTopology future with TopologyDescriptionTooLargeException, and stop returning requiresTopologyPush=true for the same (groupId, topologyEpoch) pair once that pair has been confirmed too large.
• Throttle re-solicitation without relying on the heartbeat interval. For each (groupId, groupCreationTimeMs, topologyEpoch) tuple the plugin should track (i) whether a push is currently in flight and (ii) the time of the last requiresTopologyPush=true. While a push is in flight requiresTopologyPush returns false. Once the push has completed successfully it returns false permanently for that tuple. On a transient setTopology failure the plugin arranges for requiresTopologyPush to return true again after a self-driven back-off (e.g. exponential, starting at 1s) — this is the client re-solicitation mechanism, and it is the only retry pathway: the wire-level TOPOLOGY_DESCRIPTION_UPDATE_FAILED is terminal from the client's perspective and pairs with this plugin-side re-solicitation. On permanent failure (TOPOLOGY_DESCRIPTION_TOO_LARGE or plugin-semantic INVALID_REQUEST) the plugin returns false permanently for the tuple and logs.
• Not request repeated pushes from stable groups that have already pushed their topology at the current epoch.

See the appendix for an example state machine for managing the lifecycle of a topology description.

Security Considerations

The topology description contains user-defined processor names, state-store names, and topic names. This KIP does not treat these as inherently sensitive and does not introduce any client-side redaction. Operators who consider node, store, or topic names sensitive should scope DESCRIBE ACLs on the GROUP resource accordingly: the same ACL that guards StreamsGroupDescribe guards the TopologyDescription field on the response. The UpdateStreamsGroupTopologyDescription path is guarded by READ on the GROUP, identical to the existing heartbeat ACL.

...

KIP-714 supports compression (ZStd, LZ4, GZip, Snappy) for telemetry payloads because metrics are pushed repeatedly at high frequency. Topology descriptions are pushed infrequently (only on topology epoch changes) and the wire payload is expected to fit comfortably for typical applications. The Kafka protocol's flexible versions already provide efficient serialization. Adding compression would add complexity without meaningful benefit for this use case.

Appendix: Example plugin lifecycle

Image Added

Appendix: Example RPC payload

...