...
We create one more checkpoint file, named "replication-low-watermark-checkpoint", in every log directory. The checkpoint file will have the same format as existing checkpoint files (e.g. replication-offset-checkpoint) which map TopicPartition to Long.
Proposed Changes
The idea is to add new APIs in Admin Client (see KIP-4) that can be called by user to purge data that is no longer needed. New request and response needs to be added to communicate this request between client and broker. Given the impact of this API on the data, the API should be protected by Kafka’s authorization mechanism described in KIP-11 to prevent malicious or unintended data deletion. Furthermore, we adopt the soft delete approach because it is expensive to purge data in the middle of a segment. Those segments whose maximum offset < offset-to-purge can be deleted safely. Brokers can increment low_watermark of a partition above offset-to-purge so that data with offset < offset-to-purge will not be exposed to consumer even if it is still on the disk. And the low_watermark will be checkpointed periodically similar to high_watermark to be persistent.
...
- If
low_watermark
of this partition on all followers have increased above the cutoff offsetis larger than or equal to the offsetToPurge, the result of this partition will be thelow_watermark
of the leader replica. - If
logEndOffset
of this partition is smaller than the offsetToPurge, the result of this partition will beOffsetOutOfRangeException
. - If any replica of this partition goes offline, the result of this partition will be
NotEnoughReplicasException
- If the leadership of this partition moves to another broker, the result of this partition will be
NotLeaderException
- If the result of this partition is not available after
RequestTimeoutMs
, the result of this partition will beTimeoutException
...
Given the potential damage that can be caused if this API is used by mistake, it is important that we limit its usage to only authorized users. For this matter, we can take advantage of the existing authorization framework implemented in KIP-11. purgeDataBefore()
will have the same authorization setting as deleteTopic()
. Its operation type is be DELETE and its resource type is TOPIC.
4) ListOffsetRequest
low_watermark
of a partition will be used to decide the smallest offset of the partition that will be exposed to consumer. It will returned when smallest_offset option is used in the ListOffsetRequest.
Compatibility, Deprecation, and Migration Plan
...