THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
- Kafka has topics, and topics have numbered partitions starting from 0. A topic can be created at runtime just by writing to it, but the number of partitions per topic is determined by broker configuration.
- Kafka stores messages on disk, in a series of large, append-only log files broken up into segments. Each topic+partition is a directory of these segment files. For more details, see What are Segment Files.
- An offset is just the byte offset in a given log for a topic+partition. The messages don't have any other unique identifier. They're simply stored back to back in the segment files, and you ask for them by their byte offset.
- When producing messages, the driver has to specify what topic and partition to send the message to. When requesting messages, the driver has to specify what topic, partition, and offset it wants them pulled from.
- While you can request "old" messages if you know their topic, partition, and offset, Kafka does not have a message index. You cannot efficiently query Kafka for the N-1000th message, or ask for all messages written between 30 and 35 minutes ago.
- Kafka tends to do the simplest thing possible and relies on smarter clients to keep bookkeeping. The broker does not keep track of what the client has read. More advanced setups use ZooKeeper to help with this tracking, but that is currently beyond the scope of this document.
- The protocol is a work in progress, and new point releases can introduce backwards incompatibile changes.
- The broker runs on port 9092 by default.
- If you are testing with multiple brokers on the same machine, you'll need to change both the port that they listen on (in the config file), as well as the JMX port they use. To specify a different JMX port, set the environment property JMX_PORT when starting Kafka.
Basic Objects
| Code Block | ||||
|---|---|---|---|---|
| ||||
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| REQUEST_LENGTH |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| REQUEST_TYPE | TOPIC_LENGTH |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
/ TOPIC (variable length) /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PARTITION |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
REQUEST_LENGTH = int32 // Length in bytes of entire request (excluding this field)
REQUEST_TYPE = int16 // See table below
TOPIC_LENGTH = int16 // Length in bytes of the topic name
TOPIC = String // Topic name, ASCII, not null terminated
// This becomes the name of a directory on the broker, so no
// chars that would be illegal on the filesystem.
PARTITION = int32 // Partition to act on. Number of available partitions is
// controlled by broker config. Partition numbering
// starts at 0.
============ ===== =======================================================
REQUEST_TYPE VALUE DEFINITION
============ ===== =======================================================
PRODUCE 0 Send a group of messages to a topic and partition.
FETCH 1 Fetch a group of messages from a topic and partition.
MULTIFETCH 2 Multiple FETCH requests, chained together
MULTIPRODUCE 3 Multiple PRODUCE requests, chained together
OFFSETS 4 Find offsets before a certain time (this can be a bit
misleading, please read the details of this request).
============ ===== =======================================================
|
...