Status

Current state: Under Discussion

Discussion thread: here

JIRA: here [Change the link from KAFKA-1 to your own ticket]

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

KIP-482 added support optional tagged fields. Tagged fields require no space when they're omitted from RPCs using flexible versions.
A 16 bit ErrorCode  is used in the vast majority of RPCs and will normally have value 0 (NONE). Similarly ErrorMessage will in these cases be null, but still requires a byte to store the string length.

By changing these mandatory fields to tagged fields we can save two bytes for every error code and 1 byte for every message.
Since some RPCs have error codes and messages for each batched entity in a response the saving can be more significant across a whole message.

When the error code is nonzero a tagged field will require 4 bytes (tag and size each 1 byte varints and the 16 bit value requires two bytes), rather than the 2 bytes needed for an int16.
Similarly non-null error messages will require 2 bytes (tag and size each 1 byte varints) more than the usual serialization of the string.

This KIP proposes to use tagged fields for error codes and messages in almost all RPCs in order to optimize the usual case.

Public Interfaces

Change every RPC which has an ErrorCode (except SASL_HANDSHAKE, due to KAFKA-9577) as follows:

• Increment the RPC version (both request and response), since we're making a previously required field optional
• If the RPC is not using flexibleVerions then add it (just OFFSET_DELETE).
• Add taggedVerions to all ErrorCode and ErrorMessage fields.

Proposed Changes

Compatibility, Deprecation, and Migration Plan

This change is backward compatible with existing clients.

Rejected Alternatives

Doing this for each RPC organically. This would mean a (long) period of inconsistency in how these fields are handled, would complicate other KIPs and would be easily forgotten about.