THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
We have implemented the described changes and deployed it in various setups.
Public Interfaces
Configuration Changes
Two configuration fields are added
- delete.topic.partition.enable is a boolean value indicating wether this functionality is enabled.
- delete.topic.partition.interval.delay.ms is the check interval during delayed deletion. The default value is 300000 (5 minutes).
Metadata Changes
We propose to modify PartitionRecord and add a DeleteTopicPartitionRecord in __cluster_metadata
PartitionRecord
We propose to add new field for a partition that indicates if a partition is to be removed. specifically at any time a partition can be.
ReadWrite (code=0)：it means the partition can be read from and written to.
ReadOnly (code=1)：it means the partition can only be read
None (code=-1)：it means the partition should be filtered and not written to, but consumption is not impacted.
| Code Block | ||
|---|---|---|
| ||
{
"apiKey": 3,
"type": "metadata",
"name": "PartitionRecord",
"validVersions": "0",
"fields": [
{ "name": "PartitionId", "type": "int32", "versions": "0+", "default": "-1",
"about": "The partition id." },
{ "name": "TopicId", "type": "uuid", "versions": "0+",
"about": "The unique ID of this topic." },
{ "name": "Replicas", "type": "[]int32", "versions": "0+",
"about": "The replicas of this partition, sorted by preferred order." },
{ "name": "Isr", "type": "[]int32", "versions": "0+",
"about": "The in-sync replicas of this partition" },
{ "name": "RemovingReplicas", "type": "[]int32", "versions": "0+", "nullableVersions": "0+",
"about": "The replicas that we are in the process of removing." },
{ "name": "AddingReplicas", "type": "[]int32", "versions": "0+", "nullableVersions": "0+",
"about": "The replicas that we are in the process of adding." },
{ "name": "Leader", "type": "int32", "versions": "0+", "default": "-1",
"about": "The lead replica, or -1 if there is no leader." },
{ "name": "LeaderEpoch", "type": "int32", "versions": "0+", "default": "-1",
"about": "An epoch that gets incremented each time we change the ISR." },
{ "name": "StatusMode", "type": "int32", "versions": "0+", "default": "0",
"about": "The status code of current partition." }
]
}
|
DeleteTopicPartitionRecord
| Code Block | ||
|---|---|---|
| ||
{
"apiKey": xxx,
"type": "metadata",
"name": "DeleteTopicPartitionsRecord",
"validVersions": "0",
"fields": [
{ "name": "TopicId", "type": "uuid", "versions": "0+",
"about": "The topic partition to remove. " },
{ "name": "DeletePartitions", "type": "int32", "versions": "0+","about": "The partition to remove. All associated partitions will be removed as well." },
{ "name": "DeletePartitionsDelayTimestamp", "type": "bool", "versions": "0+","about": "Delay timestamp deletion of data." },
{ "name": "CreateTimestamp", "type": "int64", "versions": "0+","about": "The record create timestamp." },
]
}
|
Protocol API changes
A new API type DeletePartitions will be added: DeletePartitionsRequest and DeletePartitionsResponse
...