DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
| Table of Contents |
|---|
Status
Current state: Under DiscussionReleased in 4.1
Discussion thread: https://lists.apache.org/thread/gyvfrysoodymgxgwnh4x2g2fs6y5247g
Vote thread: https://lists.apache.org/thread/fb5xxw9313vrljsfqsp2b2cd6gkfldjoDiscussion thread: TODO
JIRA:
| Jira | ||||||
|---|---|---|---|---|---|---|
|
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
The FETCH RPC is used to replicate both the MemoryRecords and high-watermark (HWM) for the partition. The FETCH response contains the latest HWM known by the leader and any bytes written to the partition after the FetchOffset in the FETCH request. In KRaft all of the replicas fetch uncommitted data. When the FetchOffset in the FETCH request is equal to the log end offset (LEO), KRaft parks those requests until new data is available. Once the active controller appends new records, KRaft completes all of the parked requests with the new appended records.
KRaft is not able to complete FETCH RPC based on HWM because the KRaft leader doesn't know what HWM was replicated to the remove replicas. Because of this, if the HWM changes but no new data is append HWM replica can be delayed by at most 500ms. 500ms is the maximum amount of time that the KRaft leader will park FETCH requests if there are no records after the FetchOffset.
This KIP proposes adding the replica's HWM to the FETCH request so that the KRaft leader can complete any FETCH request in which the current HWM doesn't match the replica's HWM.
Public Interfaces
Fetch
Request
Add a version 18 of the RPC. Add the field HighWatermark which the replica sets to is known high-watermark. Set to -1 if the high-watermark is not known (TODO: check that kraft users -1 on the leader when the high-watermark is not known).
Cluster Metadata and KRaft
Many users have observed that their admin operations can take up to 500 milliseconds, after they complete, to show up in other clients. See
| Jira | ||||||
|---|---|---|---|---|---|---|
|
| Jira | ||||||
|---|---|---|---|---|---|---|
|
The FETCH RPC is used to replicate both the records and the HWM for the partition. The FETCH response contains the latest HWM known by the leader and any bytes written to the partition after the FetchOffset in the FETCH request. In KRaft all of the replicas fetch uncommitted data. When the FetchOffset in the FETCH request is equal to the log end offset (LEO), KRaft parks those requests until new data is available. Once the active controller appends new records, KRaft completes all of the parked requests with the new appended records.
KRaft is not able to determine if it should park FETCH requests based on HWM. The KRaft leader doesn't know what HWM was replicated to the remote replicas. Because of this, if the HWM has changed and there is no new data after the FetchOffset, KRaft parks the FETCH request delaying the HWM replication by at most 500ms. 500ms is the maximum amount of time that the KRaft leader will park FETCH requests if there are no records after the FetchOffset.
Public Interfaces
This KIP proposes adding the replica's HWM to the FETCH request so that the replica receiving the FETCH request can use the remote replica's HWM to determine if FETCH requests should be completed immediately or parked until there is new data or a new HWM.
Metadata Version
A new metadata version will be added. The replica fetcher with send the new FETCH RPC version (18) when the metadata version has been finalized to this new metadata version.
Fetch
Request
Add a version 18 of the RPC. Add the field HighWatermark which the replica sets to its known HWM. Set to -1 if the HWM is not known. The default value for the HWM is the largest int64 or 9223372036854775807.
The remote HWM can be in the following states:
- The remote replica doesn't know the HWM. The value of the HighWatermark field is -1.
- The remote replica's latest known HWM. The value of the HighWatermark field is between 0 and 9223372036854775807, inclusive.
- The remote replica doesn't support version 18 of the FETCH RPC. The value seen by the receiving replica will be 9223372036854775807.
For the remote replicas (followers and consumer) that don't support or set the HighWatermark field, Kafka should behave the same as it does today. In other words, Kafka should not consider the HighWatermark field when determining wether to complete the FETCH request or park the FETCH request. Setting the default value for the HighWatermark field to 9223372036854775807 achieves that because the receiving replica can use the predicate "local HWN <= remote HWM" to determine if it may park the fetch request.
| Code Block |
|---|
{
"apiKey": 1,
"type": "request",
"listeners": ["broker", "controller"],
"name": "FetchRequest",
"validVersions": "4-18",
"flexibleVersions": "12+",
"fields": [
|
| Code Block |
{ "apiKey": 1, "type": "request", "listeners": ["broker", "controller"], "name": "FetchRequest", "validVersions": "4-18", "flexibleVersions": "12+", "fields": [ { "name": "ClusterId", "type": "string", "versions": "12+", "nullableVersions": "12+", "default": "null", "taggedVersions": "12+", "tag": 0, "ignorable": true, "about": "The clusterId if known. This is used to validate metadata fetches prior to broker registration." }, { "name": "ReplicaId", "type": "int32", "versions": "0-14", "default": "-1", "entityType": "brokerId", "about": "The broker ID of the follower, of -1 if this request is from a consumer." }, { "name": "ReplicaStateClusterId", "type": "ReplicaStatestring", "versions": "1512+", "taggedVersionsnullableVersions": "1512+", "tagdefault": 1"null", "abouttaggedVersions": "12+"The state of the replica in the follower.", "fields": [ , "tag": 0, "ignorable": true, "about": "The clusterId if known. This is used to validate metadata fetches prior to broker registration." }, { "name": "ReplicaId", "type": "int32", "versions": "15+0-14", "default": "-1", "entityType": "brokerId", "about": "The replicabroker ID of the follower, orof -1 if this request is from a consumer." }, { "name": "ReplicaEpochReplicaState", "type": "int64ReplicaState", "versions": "15+", "defaulttaggedVersions": "-115+", "tag": 1, "about": "The epochstate of thisthe follower,replica orin -1 if not available." }the follower.", "fields": [ ]}, { { "name": "MaxWaitMsReplicaId", "type": "int32", "versions": "0+", 15+", "default": "-1", "entityType": "brokerId", "about": "The maximum time in milliseconds to wait for the response replica ID of the follower, or -1 if this request is from a consumer." }, { "name": "MinBytesReplicaEpoch", "type": "int32int64", "versions": "0+15+", "default": "-1", "about": "The epoch minimum bytes to accumulate in the response." of this follower, or -1 if not available." } ]}, { "name": "MaxBytesMaxWaitMs", "type": "int32", "versions": "30+", "defaultabout": "0x7fffffff", "ignorable": true, "about": "The maximum bytestime toin fetch.milliseconds to See KIP-74wait for cases where this limit may not be honoredthe response." }, { "name": "IsolationLevelMinBytes", "type": "int8int32", "versions": "40+", "default": "0", "ignorable": true, "about": "ThisThe settingminimum controlsbytes theto visibilityaccumulate ofin transactionalthe recordsresponse." Using READ_UNCOMMITTED (isolation_level = 0) makes all records visible. With READ_COMMITTED (isolation_level = 1), non-transactional and COMMITTED transactional records are visible. To be more concrete, READ_COMMITTED returns all data from offsets smaller than the current LSO (last stable offset), and enables the inclusion of the list of aborted transactions in the result, which allows consumers to discard ABORTED transactional records." }, { "name": "SessionId}, { "name": "MaxBytes", "type": "int32", "versions": "3+", "default": "0x7fffffff", "ignorable": true, "about": "The maximum bytes to fetch. See KIP-74 for cases where this limit may not be honored." }, { "name": "IsolationLevel", "type": "int32int8", "versions": "74+", "default": "0", "ignorable": true, "about": "TheThis fetchsetting sessioncontrols ID." }, { "name": "SessionEpoch", "type": "int32", "versions": "7+", "default": "-1", "ignorable": true, "about": "The fetch session epoch, which is used for ordering requests in a session." }, { "name": "Topics", "type": "[]FetchTopic", "versions": "0+", "about": "The topics to fetch.", "fields": [ the visibility of transactional records. Using READ_UNCOMMITTED (isolation_level = 0) makes all records visible. With READ_COMMITTED (isolation_level = 1), non-transactional and COMMITTED transactional records are visible. To be more concrete, READ_COMMITTED returns all data from offsets smaller than the current LSO (last stable offset), and enables the inclusion of the list of aborted transactions in the result, which allows consumers to discard ABORTED transactional records." }, { "name": "TopicSessionId", "type": "stringint32", "versions": "0-127+", "entityTypedefault": "topicName0", "ignorable": true, "about": "The namefetch ofsession the topic to fetch." ID." }, { "name": "TopicIdSessionEpoch", "type": "uuidint32", "versions": "137+", "default": "-1", "ignorable": true, "about": "The uniquefetch topic ID."}, { "name": "Partitionssession epoch, which is used for ordering requests in a session." }, { "name": "Topics", "type": "[]FetchPartitionFetchTopic", "versions": "0+", "about": "The partitionstopics to fetch.", "fields": [ { "name": "PartitionTopic", "type": "int32string", "versions": "0+-12", "entityType": "topicName", "ignorable": true, "about": "The partitionname index." }, of the topic to fetch." }, { "name": "CurrentLeaderEpochTopicId", "type": "int32uuid", "versions": "913+", "default": "-1", "ignorable": true, "about": "The currentunique leader epoch of the partitiontopic ID." }, { "name": "FetchOffsetPartitions", "type": "int64[]FetchPartition", "versions": "0+", "about": "The messagepartitions to offsetfetch." },, "fields": [ { "name": "LastFetchedEpochPartition", "type": "int32", "versions": "12+", "default": "-1", "ignorable": false0+", "about": "The epochpartition of the last fetched record or -1 if there is none."}, { "index." }, { "name": "LogStartOffsetCurrentLeaderEpoch", "type": "int64int32", "versions": "59+", "default": "-1", "ignorable": true, "about": "The earliestcurrent availableleader offsetepoch of the follower replica. The field is only used when the request is sent by the follower."partition." }, { "name": "PartitionMaxBytesFetchOffset", "type": "int32int64", "versions": "0+", "about": "The maximum bytes to fetch from this partition. See KIP-74 for cases where this limit may not be honoredmessage offset." }, { "name": "ReplicaDirectoryIdLastFetchedEpoch", "type": "uuidint32", "versions": "1712+", "taggedVersionsdefault": "17+-1", "tagignorable": 0false, "ignorable": true, "about": "The directoryepoch id of the follower fetching." last fetched record or -1 if there is none."}, { "name": "HighWatermarkLogStartOffset", "type": "int64", "versions": "185+", "default": "9223372036854775807", taggedVersions": 18+-1", "tag": 1, "ignorable": true, "about": "The high-watermark known byearliest available offset of the follower replica." } ]} ]}, The field is only used when the request is sent by the follower."}, { "name": "ForgottenTopicsDataPartitionMaxBytes", "type": "[]ForgottenTopicint32", "versions": "70+", "ignorable": false, "about": "InThe anmaximum incremental fetch request, the partitions to remove.", "fields": [ bytes to fetch from this partition. See KIP-74 for cases where this limit may not be honored." }, { "name": "TopicReplicaDirectoryId", "type": "stringuuid", "versions": "7-1217+", "entityTypetaggedVersions": "topicName17+", "tag": 0, "ignorable": true, "about": "The topic name directory id of the follower fetching." }, { "name": "TopicIdHighWatermark", "type": "uuidint64", "versions": "1318+", "ignorabledefault": true"9223372036854775807", "abouttaggedVersions": 18+"The unique topic ID."}, "tag": 1, "ignorable": true, { "name": "Partitions", "typeabout": "[]int32", "versions": "7+", "about": "The partitions indexes to forgetThe high-watermark known by the replica. -1 if the high-watermark is not known." } ]}, ]}, { "name": "RackIdForgottenTopicsData", "type": "string[]ForgottenTopic", "versions": "117+", "defaultignorable": ""false, "ignorableabout": true, "about": "Rack ID of the consumer making this request."} ] } |
Response
No schema changes to the response. The version will be increase to 18 to match the request version.
Handling
TODO
Proposed Changes
TODO
Compatibility, Deprecation, and Migration Plan
TODO
Test Plan
Describe in few sentences how the KIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?
Rejected Alternatives
"In an incremental fetch request, the partitions to remove.", "fields": [
{ "name": "Topic", "type": "string", "versions": "7-12", "entityType": "topicName", "ignorable": true,
"about": "The topic name." },
{ "name": "TopicId", "type": "uuid", "versions": "13+", "ignorable": true, "about": "The unique topic ID."},
{ "name": "Partitions", "type": "[]int32", "versions": "7+",
"about": "The partitions indexes to forget." }
]},
{ "name": "RackId", "type": "string", "versions": "11+", "default": "", "ignorable": true,
"about": "Rack ID of the consumer making this request."}
]
} |
Response
No schema changes to the response. The version will be increase to 18 to match the request version.
Handling
KRaft
The current implementation always includes the HWM in the FETCH response, this will stay true in this KIP. What this KIP changes is when the FETCH request may be parked and when the FETCH request may be completed when parked.
If the response is empty and the remote replica's HWM is greater than or equal to the leader's HWM, the FETCH request is parked.
When the leader's HWM changes, the leader may complete all parked FETCH requests.
Sending
When sending FETCH requests the replica will include its HWM in the FETCH request's HighWatermark field. If the remote replica support version 18 of the RPC the HWM will be serialized. If the remote replica doesn't support version 18 the HighWatermark field in the field will ignored and not serialized.
Replica Fetcher and Kafka Consumers
Kafka consumer and the replica fetcher should not set the HighWatermark field. The broker and replica manager will ignore the HighWatermark field if specified.
Compatibility, Deprecation, and Migration Plan
Replicas that do not support version 18 of FETCH will send the previous version without the HighWatermark field. The default value for the HighWatermark field will be the maximum value of int64 (9223372036854775807). This means that from the leader's point of view the remote replica's HWM will always be greater that its HWM so the leader will never complete a FETCH request because of the value HWM. This is the behavior prior to this KIP.
If the remote replica supports version 18 of FETCH but the leader does not, the HighWatermark field will be ignored. The leader will handle the FETCH request as implemented in previous versions of Kafka.
Test Plan
For KRaft, this feature will mainly be tested using the KRaft protocol unittests as supported in KafkaRaftClientFetchTestIf there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.