Status

Current state: Under DiscussionAccepted

Discussion thread: here

JIRA: 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

It would be tedious to update the JSON message specifications to add tagged fields to each structure.  Similarly, we don't wan want to manually annotate each string, buffer, or array that should now be serialized in a more efficient way.  Instead, we should simply have the concept of "flexible versions." Any version of a message that is a "flexible version" has the changes described above.

RequestHeader Version 1

In order to have flexible version support across all requests and responses, we will bump the version of all requests and responses.  The new versions will be flexible.  (This version bump may be implemented earlier for some message types than others, depending on implementation considerations.)

RequestHeader Version 1

Requests within a "flexible version" will have a new version of Requests within a "flexible version" will have a new version of the request header.  The new RequestHeader version will be version 1, superseding version 0.  In this new version, the RequestHeader's ClientId string will be a COMPACT_STRING rather than  STRING.  Additionally, the header will contain space for tagged fields at the end.  Supporting tagged fields in the request header will give us a natural place to put additional information that is common to all requests.

...

Each optional field has a 31-bit tag number. This number must be unique within the context it appears in.  For example, we could use the tag number "1" both at the top level and within a particular substructure without creating ambiguity, since the contexts are separate.Optional   Tagged fields can have any type, except that they cannot be arrays.  Note that the restriction against having tagged arrays is just to simplify serialization.  We can relax this restriction in the future without changing the protocol on the wire.

Here is an example of a message spec which has tagged fields at both the top level and the array level:

{
  "apiKey": 9000,
  "type": "response",
  "name": "FooResponse",
  "validVersions": "0-9",
  "flexibleVersions": "9+",
  "optionalFieldsfields": [
    {
      { "name": "UserAgent", "type": "string", "tag": 0, "taggedVersions": "9+",
        "about": "The user-agent that sent this request." },
  ],
  "fields": [
    { "name": "Foos", "type": "[]Foo", "versions": "0+",
      "about": "Each foo.", "optionalFieldsfields": [
        { "name": "Bar", "type": "string", "tag": 0": 0, "taggedVersions": "9+",
          "default": "hello world", "about": "The bar associated with this foo, if any." },
      ], "fields": [
        { "name": "Baz", "type": "int16", "versions": "0+",
          "about": "The baz associated with this foo." },
  ...
  ]
}

Type Classes

...

Type Class Name	Field Class Name	Description
CompactArrayOf	CompactArray	Represents an array whose length is expressed as a variable-length integer rather than a fixed 4-byte length.
COMPACT_STRING	CompactString	Represents a string whose length is expressed as a variable-length integer rather than a fixed 2-byte length.
COMPACT_NULLABLE_STRING	CompactNullableString	Represents a nullable string whose length is expressed as a variable-length integer rather than a fixed 2-byte length.
COMPACT_BYTES	CompactBytes	Represents a byte buffer whose length is expressed as a variable-length integer rather than a fixed 4-byte length.
COMPACT_NULLABLE_BYTES	CompactNullableBytes	Represents a nullable byte buffer whose length is expressed as a variable-length integer rather than a fixed 4-byte length.
TaggedFieldsTaggedFields	TaggedFieldsSection	Represents a section containing optional tagged fields.

...