Status

Current state: Under Discussion Accepted

Discussion thread: here 

JIRA: here 

...

{
  "apiKey": 28,
  "type": "response",
  "name": "TxnOffsetCommitResponse",
  // ...
  // Version 6 adds support for topic IDs and removes support for
  // topic names. It can also return GROUP_ID_NOT_FOUND and
  // STALE_MEMBER_EPOCH (KIP-1319).
  "validVersions": "0-6",                                    // CHANGED
  "flexibleVersions": "3+",
  // Supported errors:
  // ...
  // - GROUP_ID_NOT_FOUND (version 6+)                       // NEW
  // - STALE_MEMBER_EPOCH (version 6+)                       // NEW
  // - UNKNOWN_TOPIC_ID (version 6+)                         // NEW
  "fields": [
    { "name": "ThrottleTimeMs", "type": "int32", "versions": "0+" },
    { "name": "Topics",
      "type": "[]TxnOffsetCommitResponseTopic", "versions": "0+",
      "about": "The responses for each topic.", "fields": [
        { "name": "Name", "type": "string",
          "versions": "0-5",                                  // CHANGED
          "entityType": "topicName", "ignorable": true,
          "about": "The topic name." },
        { "name": "TopicId", "type": "uuid",                  // NEW
          "versions": "6+", "ignorable": true,
          "about": "The topic ID." },
        { "name": "Partitions",
          "type": "[]TxnOffsetCommitResponsePartition",
          "versions": "0+", "fields": [
            { "name": "PartitionIndex", "type": "int32", "versions": "0+" },
            { "name": "ErrorCode", "type": "int16", "versions": "0+" }
          ]
        }
      ]
    }
  ]
}

New error codes:|

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

For v0-5, all existing error mappings are preserved.

Version 6 of the API supports the following new error code:

- `GROUP_ID_NOT_FOUND` - The group does not exist. Previously returned as `ILLEGAL_GENERATION`.
- `STALE_MEMBER_EPOCH` - The member epoch is stale under the new consumer group protocol. Previously returned as `ILLEGAL_GENERATION`.
- `UNKNOWN_TOPIC_ID` - The provided topic ID cannot be resolved to a known topic.

Proposed Changes

Broker

When the broker receives a v6 request, it resolves topic IDs to topic names and returns `UNKNOWN_TOPIC_ID` for unresolvable IDs. The resolved names are used for authorization, partition validation, and offset storage. The topic ID is persisted in the `OffsetCommitValue` record, replacing the hardcoded `Uuid.ZERO_UUID`. For v6+ requests, the broker returns `GROUP_ID_NOT_FOUND` and `STALE_MEMBER_EPOCH` directly instead of mapping them to `ILLEGAL_GENERATION` when the consumer rebalance protocol is used. Existing behavior for v0-5 is preserved.

Producer

The sendOffsetsToTransaction API remains unchanged. It accepts a Map<TopicPartition, OffsetAndMetadata> and a ConsumerGroupMetadata. When sendOffsetsToTransaction is called, the producer fetches metadata for the topics in the offsets map and waits for the response. This ensures topic IDs are available for topics the producer has not previously produced to. Once metadata is resolved, if all topics have a topic ID, the producer builds a v6 request. If any topic lacks a topic ID, it falls back to v5. The producer treats `STALE_MEMBER_EPOCH` and `GROUP_ID_NOT_FOUND` as abortable errors and `UNKNOWN_TOPIC_ID` as a fatal error.

Compatibility, Deprecation, and Migration Plan

The producer automatically negotiates the version based on broker support and topic ID availability. `LAST_STABLE_VERSION_BEFORE_TRANSACTION_V2 = 4` caps non-TV2 clients at v4, so v6 effectively requires Transaction V2 (KIP-890). No existing versions are deprecated.

Test Plan

The changes are covered by unit tests, integration tests, and system tests.

Rejected Alternatives

...

Proposed Changes

Broker-Side

• Topic ID resolution. For v6 requests, the broker resolves topic IDs to names via the metadata cache before authorization and partition validation. This follows the same pattern as OffsetCommit v10. The per-partition error code returned for an invalid topic is determined in this order:
  • If the topic ID cannot be resolved by the broker, return UNKNOWN_TOPIC_ID for all partitions of that topic.
  • If the client is not authorized for the resolved topic name, return TOPIC_AUTHORIZATION_FAILED for all partitions of that topic.
  • If a requested partition does not exist, return UNKNOWN_TOPIC_OR_PARTITION for that partition only.
• Error mapping. For v6+, GROUP_ID_NOT_FOUND and STALE_MEMBER_EPOCH are returned directly instead of being mapped to ILLEGAL_GENERATION. For v0-5, the existing mapping is preserved.
• Persisting topic IDs. The broker populates the existing topicId tagged field in OffsetCommitValue v4 with the real topic ID instead of Uuid.ZERO_UUID. No new record schema is introduced.

Producer-Side

• sendOffsetsToTransaction API unchanged. The public API does not change.
• Topic ID resolution. When sendOffsetsToTransaction  is called and the broker supports v6, the producer adds the topics from the offset map to its metadata cache and waits for a refresh (bounded by max.block.ms) before building the request with the resolved topic IDs. No explicit cleanup is performed: the producer's metadata cache already evicts topics that have been idle for longer than metadata.max.idle.ms (default 5 minutes), which gives transient semantics for free. Topics that are committed repeatedly stay cached, which avoids a metadata wait on subsequent commits.
• Fallback. If the broker supports v6 but any topic in the offset map has an unavailable topic ID, the producer falls back to v5 for that request (all-or-nothing, same pattern as OffsetCommit's canUseTopicIds). In practice, this should never trigger since Transaction V2 requires KRaft, which always assigns topic IDs.
• Error handling. GROUP_ID_NOT_FOUND and STALE_MEMBER_EPOCH are treated as abortable errors. The application must abort the transaction. UNKNOWN_TOPIC_ID is treated as a retriable error consistent with UNKNOWN_TOPIC_OR_PARTITION. Since UnknownTopicIdException is a RetriableException, it falls into the existing retriable error handling path: the partition stays in pendingTxnOffsetCommits and the handler re-enqueues the request.
• Response mapping. For v6 responses (keyed by topic ID), the handler uses the topic ID to name mapping built during request construction to correlate responses back to TopicPartition.

Compatibility, Deprecation, and Migration Plan

• Version negotiation. v0-4 require no Transaction V2. v5 requires Transaction V2. v6 requires Transaction V2 and uses topic IDs with a fallback to v5.
• Rolling upgrades. No new record schema is introduced. The existing OffsetCommitValue v4 topicId tagged field (`ignorable: true`) is populated with real topic IDs by new brokers and ZERO_UUID by old brokers.
• Backward compatibility. Clients using v0-5 are unaffected. Brokers that do not support v6 negotiate a lower version.

Test Plan

The changes are covered by unit tests, integration tests, and system tests.

Rejected Alternatives

• Splitting Into Two Versions (Error Codes + Topic IDs). OffsetCommit split error codes (v9) and topic IDs (v10) because v9 could be used without KRaft. For TxnOffsetCommit, v5+ already requires Transaction V2 (which requires KRaft), so any broker
  supporting v6 necessarily supports both capabilities. Splitting would add protocol complexity without practical benefit.
• Keeping Topic Names Alongside Topic IDs in v6. The established Kafka convention s a clean break: Name in versions 0-N, TopicId in N+1 onward. Sending both adds redundancy and ambiguity.