Status

Current state: Under DiscussionAccepted

Discussion thread: here

JIRA: 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

It would be tedious to update the JSON message specifications to add tagged fields to each structure.  Similarly, we don't wan want to manually annotate each string, buffer, or array that should now be serialized in a more efficient way.  Instead, we should simply have the concept of "flexible versions." Any version of a message that is a "flexible version" has the changes described above.

RequestHeader Version 1

In order to have flexible version support across all requests and responses, we will bump the version of all requests and responses.  The new versions will be flexible.  (This version bump may be implemented earlier for some message types than others, depending on implementation considerations.)

RequestHeader Version 1

Requests within a "flexible version" will have Requests within a "flexible version" will have a new version of the request header.  The new RequestHeader version will be version 1, superseding version 0.  In this new version, the RequestHeader's ClientId string will be a COMPACT_STRING rather than  STRING.  Additionally, the header will contain space for tagged fields at the end.  Supporting tagged fields in the request header will give us a natural place to put additional information that is common to all requests.

...

Responses within a "flexible version" will have a new version of the response header.  The new ResponseHeader version will be version 1, superseding version 0.  In this new version, the header will contain space for tagged fields at the end.  Supporting Supporting tagged fields in the response header will give us a natural place to put additional information that is common to all responses.

...

The flexible versions will be described by a new top-level field in each request and response.  The format will be the same as that of existing version fields.  If the The flexible versions are not specified, it is assumed that all versions are flexiblemust be specified in each JSON file.

Note that adding support for tagged versions to an RPC requires bumping the protocol version number.

...

Each optional field has a 31-bit tag number. This number must be unique within the context it appears in.  For example, we could use the tag number "1" both at the top level and within a particular substructure without creating ambiguity, since the contexts are separate.Optional   Tagged fields can have any type, except that they cannot be arrays.  Note that the restriction against having tagged arrays is just to simplify serialization.  We can relax this restriction in the future without changing the protocol on the wire.

Here is an example of a message spec which has tagged fields at both the top level and the array level:

{
  "apiKey": 9000,
  "type": "response",
  "name": "FooResponse",
  "validVersions": "0-9",
  "flexibleVersions": "9+",
  "optionalFieldsfields": [
    {
      { "name": "UserAgent", "type": "string", "tag": 0, "tagtaggedVersions": 0"9+",
        "about": "The user-agent that sent this request." },
  ],
  "fields": [
    { "name": "Foos", "type": "[]Foo", "versions": "0+",
      "about": "Each foo.", "optionalFieldsfields": [
        { "name": "Bar", "type": "string", "tag": 0, "taggedVersions": "9+",
          "default": "hello world", "about": "The bar associated with this foo, if any." },
      ], "fields": [ associated with this foo, if any." },
        { "name": "Baz", "type": "int16", "versions": "0+",
          "about": "The baz associated with this foo." },
  ...
  ]
}

Type Classes

...

Type Class Name	Field Class Name	Description
CompactArrayOf	CompactArray	Represents an array whose length is expressed as a variable-length integer rather than a fixed 4-byte length.
COMPACT_STRING	CompactString	Represents a string whose length is expressed as a variable-length integer rather than a fixed 2-byte length.
COMPACT_NULLABLE_STRING	CompactNullableString	Represents a nullable string whose length is expressed as a variable-length integer rather than a fixed 2-byte length.
COMPACT_BYTES	CompactBytes	Represents a byte buffer whose length is expressed as a variable-length integer rather than a fixed 4-byte length.
COMPACT_NULLABLE_BYTES	CompactNullableBytes	Represents a nullable byte buffer whose length is expressed as a variable-length integer rather than a fixed 4-byte length.
TaggedFieldsTaggedFields	TaggedFieldsSection	Represents a section containing optional tagged fields.

...

In a flexible version, each structure begins ends with a tag section.  This section stores all of the tagged fields in the structure.

...

All requests and responses will begin end with a tagged field buffer.  If there are no tagged fields, this will only be a single zero byte.

...