Page History

...

This document covers the protocol implemented in Kafka 0.8. It is meant to give a readable guide to the protocol that covers the available requests, their binary format, and the proper way to make use of them to implement a client. This document assumes you understand the basic design and terminology described here.

...

Overview

The Kafka uses a binary protocol over TCP. This protocol defines all apis as request response message pairs. All messages are size delimited and are made up of the following primitive types:

...

Fixed Width Primitives

...

int8, int16, int32, int64, uint8, uint16, uint32, uint64 - Integers with the given precision (in bits) stored in big endian order. unitXX variants are unsigned and have double the range.

...

Variable Length Primitives

...

protocol is fairly simple, there are only four client requests APIs.

1. Metadata - Describes the currently available brokers, their host and port information, and gives information about which broker hosts which partitions.
2. Send - Send messages to a broker
3. Fetch - Fetch messages from a broker, one which fetches data, one which gets cluster metadata, and one which gets offset information about a topic.
4. Offsets - Get information about the available offsets for a given topic partition.

Each of these will be described in detail below.

Preliminaries

Kafka uses a binary protocol over TCP. The protocol defines all apis as request response message pairs. All messages are size delimited and are made up of the following primitive types.

...

Fixed Width Primitives

...

int8, int16, int32, int64, uint8, uint16, uint32, uint64 - Integers with the given precision (in bits) stored in big endian order. unitXX variants are unsigned and have double the range.

...

Variable Length Primitives

...

bytes16, bytes32, string16 - These types bytes16, bytes32, string16 - These types consist of a signed integer giving a length N followed by N bytes of content. -1 indicates null. bytes16 and string16 use a two byte int16 size and bytes32 uses a four byte int32 size. string16 is identical in format to bytes16 but the bytes should be interpreted as UTF8 encoded characters.

...

It is often useful to repeat some structure multiple times. This will always be encoded as a int size containing the length N followed by N repetitions of the structure which can itself be made up of other primitive types. In the BNF grammars below we will show an array of a structure foo as \[foo\].

Client Requests

The Kafka protocol is fairly simple, there are only four client requests APIs.

1. Metadata - Describes the currently available brokers, their host and port information, and gives information about which broker hosts which partitions.
2. Send - Send messages to a broker
3. Fetch - Fetch messages from a broker, one which fetches data, one which gets cluster metadata, and one which gets offset information about a topic.
4. Offsets - Get information about the available offsets for a given topic partition.

Each of these will be described in detail below

.

Notes on reading the request format grammars

...