DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
Many users have observed that their admin operations can take up to 500 milliseconds after they complete to show up in other clients. See
| Jira | ||||||
|---|---|---|---|---|---|---|
|
| Jira | ||||||
|---|---|---|---|---|---|---|
|
The FETCH RPC is used to replicate both the records and the HWM for the partition. The FETCH response contains the latest HWM known by the leader and any bytes written to the partition after the FetchOffset in the FETCH request. In KRaft all of the replicas fetch uncommitted data. When the FetchOffset in the FETCH request is equal to the log end offset (LEO), KRaft parks those requests until new data is available. Once the active controller appends new records, KRaft completes all of the parked requests with the new appended records.
KRaft is not able to determine if it should park FETCH requests based on HWM because the . The KRaft leader doesn't know what HWM was replicated to the remote replicas. Because of this, if the HWM has changed and there is no new data after the FetchOffset, KRaft parks the FETCH request delay delaying the HWM replication by at most 500ms. 500ms is the maximum amount of time that the KRaft leader will park FETCH requests if there are no records after the FetchOffset.
...
When the leader's HWM changes, the leader will complete all parked FETCH requests.
Sending
When sending FETCH requests the replica will include its HWM in the FETCH request's HighWatermark field. If the remote replica support version 18 of the RPC the HWM will be serialized. If the remote replica doesn't support version 18 the HighWatermark field in the field will ignored and not serialized.
Compatibility, Deprecation, and Migration Plan
Replicas that do not support version 18 of FETCH will send the previous version with without the missing HighWatermark field. The default value for the HighWatermark field will be the maximum value of int64 (9223372036854775807). This means that from the leader's point of view the remote replica's HWM will always be greater that its HWM so the leader will never park a FETCH request because of the HWM. This is the behavior prior to this KIP.
If the remote replica supports version 18 of FETCH but the leader does not, the HighWatermark field will be ignored. The leader will handle the FETCH request as implemented in previous version versions of KRaft.
Test Plan
This feature will mainly be tested using the KRaft protocol unittests as supported in KafkaRaftClientFetchTest.