THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
| Table of Contents |
|---|
Motivation
Apache Kafka Brokers make periodic FetchRequests to other brokers, in order to learn about updates to partitions they are following. These periodic FetchRequests must enumerate all the partitions which the follower is interested in. The responses also enumerate all the partitions, plus metadata (and potentially data) about each one.
...
- max.incremental.fetch.session.cache.slots, which set the number of incremental fetch session cache slots on the broker. Default value: 101,000
There is one new constant for fetch session caching:
...
The fetch epoch keeps the state on the leader and the follower synchronized. It ensures that if a message is duplicated or lost, the server will always notice. It is also used to associate requests and responses in the logs. Other numbers, such as IP addresses and ports, or NetworkClient sequence numbers, can also be helpful for this purpose-- but they are less likely to be unique.
FetchRequest Metadata meaning
Request SessionId | Request SessionEpoch | Meaning |
|---|---|---|
| 0 | -1 |
Make a full FetchRequest that does not use or create a session. This is the session ID used by pre-KIP-227 FetchRequests. | ||
| 0 | 0 | Make a full FetchRequest. Create a new incremental fetch session if possible. If a new fetch session is created, it will start at epoch 1. |
| $ID | 0 | Close the incremental fetch session identified by $ID. Make a full FetchRequest. Create a new incremental fetch session if possible. If a new fetch session is created, it will start at epoch 1. |
| $ID | $EPOCH | If the ID and EPOCH are correct, make an incremental fetch request. |
| $ID | -1 | Close the incremental fetch session identified by $ID. Make a full FetchRequest. |
Incremental Fetch Requests
...
When the top-level error code is set, the caller should assume that all the partitions in the fetch received the given error.
Fetch Session ID
The FetchResponse now contains a 32-bit fetch session ID.
FetchResponse Metadata meaning
Request SessionId | Meaning |
|---|---|
| 0 | No fetch session was created. |
| $ID | The next request can be an incremental fetch request with the given $ID. |
Incremental Fetch Responses
...
NumIncrementalFetchSessions: Tracks the number of incremental fetch sessions which exist.
NumIncrementalFetchPartitionsCached: Tracks the total number of partitions cached by incremental fetch sessions.
NumIncrementalFetchSessionsEvictedIncrementalFetchSessionEvictionsPerSec: Tracks the number of incremental fetch sessions that were evicted from the cache per second. This metric is not increased when a client closes its own incremental fetch sessionNumIncrementalFetchPartitionsCached: Tracks the total number of partitions cached by incremental fetch sessions.
Compatibility, Deprecation, and Migration Plan
...