DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
DisklessCreatePartitions: Create new partitions, could be used both for creating new topics and for increasing the number of partitions in an existing one.DisklessDeleteTopics: Delete existing topics and their partitions.DisklessCommitFile: Commit a WAL file with batches. During this operation, various checks are performed (e.g. for producer idempotence, transactional checks) and offsets are assigned to batches.DisklessDeleteRecords: Delete tail records in partitions.DisklessListOffsets: List offsets for specified timestamps, including special timestamps for “earliest”, “latest” for finding the log start offset and high watermark. This is a read-only operation with strict consistency (i.e. it should reflect the state after applying all the previous mutating operations).DisklessFindBatches: Find batches in the specified partitions starting from the specified offset. This is a read-only operation with less strict consistency requirements (i.e. could be served from stale state and by followers).DisklessTieredStorageOffload: Commit offloading specified offsets into tiered storage as a part of a segment, allowing to delete the associated batches.DisklessEndTransaction: End transactions.DisklessCheckFile:DisklessDescribeFile: Check if a WAL file is known to the coordinator has live batches of this coordinator are stored in itin a WAL file.
The request-response protocol for these operations will be explained in the Public interfaces section.
Some operations remain out of scope of this KIP, such as the ones for transaction management or offloading to tiered storage. They will be explained in the corresponding KIPs.
Topic-based implementation
...
And last optimization but not least is that sending requests to DCs on the same broker doesn’t need to be a network call, so the upper bound is in any case n_brokers-1 or n_racks-1. This matters because DC can be its own produce gateway in its own rack.
Public Interfaces
The proposed public interface changes are subject to change during the community discussion.
Diskless coordinator API
...
DisklessCreatePartitions
...
| Code Block |
|---|
TBD
Configuration
TBD
Monitoring
| ||||
{
"apiKey": 93,
"type": "request",
"listeners": ["controller"],
"name": "DisklessCreatePartitionsRequest",
// Version 0 is the initial version.
"validVersions": "0",
"flexibleVersions": "0+",
"fields": [
{ "name": "Topics", "type": "[]DisklessCreatablePartitions", "versions": "0+",
"about": "Each topic where to create partitions.", "fields": [
{ "name": "TopicId", "type": "uuid", "versions": "0+", "mapKey": true,
"about": "The unique topic ID." },
{ "name": "Name", "type": "string", "versions": "0+", "entityType": "topicName",
"about": "The topic name." },
{ "name": "NumPartitions", "type": "int32", "versions": "0+",
"about": "The number of partitions to create for the topic." }
]},
{ "name": "TimeoutMs", "type": "int32", "versions": "0+", "default": "60000",
"about": "How long to wait in milliseconds before timing out the request." }
]
}
{
"apiKey": 93,
"type": "response",
"name": "DisklessCreatePartitionsResponse",
// Version 0 is the initial version.
"validVersions": "0",
"flexibleVersions": "0+",
"fields": [
{ "name": "ThrottleTimeMs", "type": "int32", "versions": "0+", "ignorable": true,
"about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
{ "name": "Topics", "type": "[]DisklessCreatablePartitionsResult", "versions": "0+",
"about": "Results for each topic.", "fields": [
{ "name": "TopicId", "type": "uuid", "versions": "0+", "mapKey": true,
"about": "The unique topic ID." },
{ "name": "ErrorCode", "type": "int16", "versions": "0+",
"about": "The error code, or 0 if there was no error." },
{ "name": "ErrorMessage", "type": "string", "versions": "0+", "nullableVersions": "0+", "ignorable": true,
"about": "The error message, or null if there was no error." }
]}
]
} |
DisklessDeleteTopics
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"apiKey": 94,
"type": "request",
"listeners": ["controller"],
"name": "DisklessDeleteTopicsRequest",
// Version 0 is the initial version.
"validVersions": "0",
"flexibleVersions": "0+",
"fields": [
{ "name": "Topics", "type": "[]DisklessDeletableTopicState", "versions": "0+",
"about": "Each topic to delete.", "fields": [
{ "name": "TopicId", "type": "uuid", "versions": "0+", "mapKey": true,
"about": "The unique topic ID." }
]},
{ "name": "TimeoutMs", "type": "int32", "versions": "0+", "default": "60000",
"about": "The length of time in milliseconds to wait for the deletions to complete." }
]
}
{
"apiKey": 94,
"type": "response",
"name": "DisklessDeleteTopicsResponse",
// Version 0 is the initial version.
"validVersions": "0",
"flexibleVersions": "0+",
"fields": [
{ "name": "ThrottleTimeMs", "type": "int32", "versions": "0+", "ignorable": true,
"about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
{ "name": "Responses", "type": "[]DisklessDeletableTopicResult", "versions": "0+",
"about": "The results for each topic we tried to delete.", "fields": [
{ "name": "TopicId", "type": "uuid", "versions": "0+", "mapKey": true,
"about": "The unique topic ID." },
{ "name": "ErrorCode", "type": "int16", "versions": "0+",
"about": "The deletion error, or 0 if the deletion succeeded." },
{ "name": "ErrorMessage", "type": "string", "versions": "0+", "nullableVersions": "0+", "ignorable": true, "default": "null",
"about": "The error message, or null if there was no error." }
]}
]
} |
DisklessCommitFile
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"apiKey": 95,
"type": "request",
"listeners": ["controller"],
"name": "DisklessCommitFileRequest",
// Version 0 is the initial version.
"validVersions": "0",
"flexibleVersions": "0+",
"fields": [
{ "name": "ObjectKey", "type": "string", "versions": "0+",
"about": "The object key identifying the file in object storage." },
{ "name": "ObjectFormat", "type": "int8", "versions": "0+",
"about": "The format of the object (e.g. 1 = WRITE_AHEAD_MULTI_SEGMENT)." },
{ "name": "FileSize", "type": "int64", "versions": "0+",
"about": "The total size of the file in bytes." },
{ "name": "OwnerIds", "type": "[]int32", "versions": "0+", "entityType": "brokerId",
"about": "The list of Diskless Coordinator IDs that share ownership of this file." },
{ "name": "Batches", "type": "[]DisklessCommitBatch", "versions": "0+",
"about": "The batches contained in the file to commit.", "fields": [
{ "name": "Magic", "type": "int8", "versions": "0+",
"about": "The record batch magic byte." },
{ "name": "RequestId", "type": "int32", "versions": "0+",
"about": "The ID of the original produce request this batch belongs to." },
{ "name": "TopicId", "type": "uuid", "versions": "0+",
"about": "The unique ID of the topic this batch belongs to." },
{ "name": "PartitionIndex", "type": "int32", "versions": "0+",
"about": "The partition index this batch belongs to." },
{ "name": "ByteOffset", "type": "int32", "versions": "0+",
"about": "The byte offset of this batch within the file." },
{ "name": "Size", "type": "int32", "versions": "0+",
"about": "The size of the batch in bytes." },
{ "name": "BaseOffset", "type": "int64", "versions": "0+",
"about": "The base offset of the record batch." },
{ "name": "LastOffset", "type": "int64", "versions": "0+",
"about": "The last offset of the record batch." },
{ "name": "BatchMaxTimestamp", "type": "int64", "versions": "0+",
"about": "The maximum timestamp in the batch." },
{ "name": "TimestampType", "type": "int8", "versions": "0+",
"about": "The timestamp type (0 = CreateTime, 1 = LogAppendTime)." },
{ "name": "ProducerId", "type": "int64", "versions": "0+",
"about": "The producer ID, or -1 if not set." },
{ "name": "ProducerEpoch", "type": "int16", "versions": "0+",
"about": "The producer epoch, or -1 if not set." },
{ "name": "BaseSequence", "type": "int32", "versions": "0+",
"about": "The base sequence number, or -1 if not set." },
{ "name": "LastSequence", "type": "int32", "versions": "0+",
"about": "The last sequence number, or -1 if not set." }
]},
{ "name": "TimeoutMs", "type": "int32", "versions": "0+", "default": "60000",
"about": "How long to wait in milliseconds before timing out the request." }
]
}
{
"apiKey": 95,
"type": "response",
"name": "DisklessCommitFileResponse",
// Version 0 is the initial version.
"validVersions": "0",
"flexibleVersions": "0+",
"fields": [
{ "name": "ThrottleTimeMs", "type": "int32", "versions": "0+", "ignorable": true,
"about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
{ "name": "Batches", "type": "[]DisklessCommitBatchResult", "versions": "0+",
"about": "Results for each batch, in the same order as the request batches.", "fields": [
{ "name": "ErrorCode", "type": "int16", "versions": "0+",
"about": "The error code, or 0 if there was no error." },
{ "name": "ErrorMessage", "type": "string", "versions": "0+", "nullableVersions": "0+", "ignorable": true, "default": "null",
"about": "The error message, or null if there was no error." },
{ "name": "AssignedBaseOffset", "type": "int64", "versions": "0+",
"about": "The base offset assigned by the coordinator, or -1 on error." },
{ "name": "LogAppendTime", "type": "int64", "versions": "0+",
"about": "The log append timestamp, or -1 if not applicable." },
{ "name": "LogStartOffset", "type": "int64", "versions": "0+",
"about": "The log start offset of the partition, or -1 on error." },
{ "name": "IsDuplicate", "type": "bool", "versions": "0+",
"about": "True if this batch was a duplicate of an already committed batch." }
]}
]
} |
DisklessDeleteRecords
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"apiKey": 96,
"type": "request",
"listeners": ["controller"],
"name": "DisklessDeleteRecordsRequest",
// Version 0 is the initial version.
"validVersions": "0",
"flexibleVersions": "0+",
"fields": [
{ "name": "Topics", "type": "[]DisklessDeleteRecordsTopic", "versions": "0+",
"about": "Each topic that we want to delete records from.", "fields": [
{ "name": "TopicId", "type": "uuid", "versions": "0+", "mapKey": true,
"about": "The unique topic ID." },
{ "name": "Partitions", "type": "[]DisklessDeleteRecordsPartition", "versions": "0+",
"about": "Each partition that we want to delete records from.", "fields": [
{ "name": "PartitionIndex", "type": "int32", "versions": "0+", "mapKey": true,
"about": "The partition index." },
{ "name": "Offset", "type": "int64", "versions": "0+",
"about": "The deletion offset. All records with offsets less than this value will be deleted." }
]}
]},
{ "name": "TimeoutMs", "type": "int32", "versions": "0+", "default": "60000",
"about": "How long to wait for the deletion to complete, in milliseconds." }
]
}
{
"apiKey": 96,
"type": "response",
"name": "DisklessDeleteRecordsResponse",
// Version 0 is the initial version.
"validVersions": "0",
"flexibleVersions": "0+",
"fields": [
{ "name": "ThrottleTimeMs", "type": "int32", "versions": "0+", "ignorable": true,
"about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
{ "name": "Topics", "type": "[]DisklessDeleteRecordsTopicResult", "versions": "0+",
"about": "Each topic that we wanted to delete records from.", "fields": [
{ "name": "TopicId", "type": "uuid", "versions": "0+", "mapKey": true,
"about": "The unique topic ID." },
{ "name": "Partitions", "type": "[]DisklessDeleteRecordsPartitionResult", "versions": "0+",
"about": "Each partition that we wanted to delete records from.", "fields": [
{ "name": "PartitionIndex", "type": "int32", "versions": "0+", "mapKey": true,
"about": "The partition index." },
{ "name": "LowWatermark", "type": "int64", "versions": "0+",
"about": "The partition low water mark." },
{ "name": "ErrorCode", "type": "int16", "versions": "0+",
"about": "The deletion error code, or 0 if the deletion succeeded." },
{ "name": "ErrorMessage", "type": "string", "versions": "0+", "nullableVersions": "0+", "ignorable": true, "default": "null",
"about": "The error message, or null if there was no error." }
]}
]}
]
} |
DisklessListOffsets
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"apiKey": 97,
"type": "request",
"listeners": ["controller"],
"name": "DisklessListOffsetsRequest",
// Version 0 is the initial version.
"validVersions": "0",
"flexibleVersions": "0+",
"fields": [
{ "name": "Topics", "type": "[]DisklessListOffsetsTopic", "versions": "0+",
"about": "Each topic in the request.", "fields": [
{ "name": "TopicId", "type": "uuid", "versions": "0+", "mapKey": true,
"about": "The unique topic ID." },
{ "name": "Partitions", "type": "[]DisklessListOffsetsPartition", "versions": "0+",
"about": "Each partition in the request.", "fields": [
{ "name": "PartitionIndex", "type": "int32", "versions": "0+", "mapKey": true,
"about": "The partition index." },
{ "name": "Timestamp", "type": "int64", "versions": "0+",
"about": "The timestamp to query. Use -2 for earliest, -1 for latest, -3 for max timestamp, -4 for earliest local, -5 for latest tiered." }
]}
]},
{ "name": "TimeoutMs", "type": "int32", "versions": "0+", "default": "60000",
"about": "How long to wait in milliseconds before timing out the request." }
]
}
{
"apiKey": 97,
"type": "response",
"name": "DisklessListOffsetsResponse",
// Version 0 is the initial version.
"validVersions": "0",
"flexibleVersions": "0+",
"fields": [
{ "name": "ThrottleTimeMs", "type": "int32", "versions": "0+", "ignorable": true,
"about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
{ "name": "Topics", "type": "[]DisklessListOffsetsTopicResponse", "versions": "0+",
"about": "Each topic in the response.", "fields": [
{ "name": "TopicId", "type": "uuid", "versions": "0+", "mapKey": true,
"about": "The unique topic ID." },
{ "name": "Partitions", "type": "[]DisklessListOffsetsPartitionResponse", "versions": "0+",
"about": "Each partition in the response.", "fields": [
{ "name": "PartitionIndex", "type": "int32", "versions": "0+", "mapKey": true,
"about": "The partition index." },
{ "name": "ErrorCode", "type": "int16", "versions": "0+",
"about": "The partition error code, or 0 if there was no error." },
{ "name": "ErrorMessage", "type": "string", "versions": "0+", "nullableVersions": "0+", "ignorable": true, "default": "null",
"about": "The error message, or null if there was no error." },
{ "name": "Timestamp", "type": "int64", "versions": "0+", "default": "-1",
"about": "The timestamp associated with the returned offset." },
{ "name": "Offset", "type": "int64", "versions": "0+", "default": "-1",
"about": "The returned offset." }
]}
]}
]
} |
DisklessFindBatches
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"apiKey": 98,
"type": "request",
"listeners": ["controller"],
"name": "DisklessFindBatchesRequest",
// Version 0 is the initial version.
"validVersions": "0",
"flexibleVersions": "0+",
"fields": [
{ "name": "FetchMaxBytes", "type": "int32", "versions": "0+",
"about": "Maximum bytes to fetch across all partitions." },
{ "name": "MaxBatchesPerPartition", "type": "int32", "versions": "0+",
"about": "Maximum number of batches per partition to return." },
{ "name": "Topics", "type": "[]DisklessFindBatchesTopic", "versions": "0+",
"about": "Each topic in the request.", "fields": [
{ "name": "TopicId", "type": "uuid", "versions": "0+", "mapKey": true,
"about": "The topic ID." },
{ "name": "Partitions", "type": "[]DisklessFindBatchesPartition", "versions": "0+",
"about": "Each partition in the request.", "fields": [
{ "name": "PartitionIndex", "type": "int32", "versions": "0+", "mapKey": true,
"about": "The partition index." },
{ "name": "Offset", "type": "int64", "versions": "0+",
"about": "The offset to start fetching from." },
{ "name": "MaxPartitionFetchBytes", "type": "int32", "versions": "0+",
"about": "Maximum bytes to fetch for this partition." }
]}
]},
{ "name": "TimeoutMs", "type": "int32", "versions": "0+", "default": "60000",
"about": "How long to wait in milliseconds before timing out the request." }
]
}
{
"apiKey": 98,
"type": "response",
"name": "DisklessFindBatchesResponse",
// Version 0 is the initial version.
"validVersions": "0",
"flexibleVersions": "0+",
"fields": [
{ "name": "ThrottleTimeMs", "type": "int32", "versions": "0+", "ignorable": true,
"about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
{ "name": "Topics", "type": "[]DisklessFindBatchesTopicResponse", "versions": "0+",
"about": "Each topic in the response.", "fields": [
{ "name": "TopicId", "type": "uuid", "versions": "0+", "mapKey": true,
"about": "The topic ID." },
{ "name": "Partitions", "type": "[]DisklessFindBatchesPartitionResponse", "versions": "0+",
"about": "Each partition in the response.", "fields": [
{ "name": "PartitionIndex", "type": "int32", "versions": "0+", "mapKey": true,
"about": "The partition index." },
{ "name": "ErrorCode", "type": "int16", "versions": "0+",
"about": "The partition error code, or 0 if there was no error." },
{ "name": "LogStartOffset", "type": "int64", "versions": "0+", "default": "-1",
"about": "The log start offset of the partition, or -1 if unknown." },
{ "name": "HighWatermark", "type": "int64", "versions": "0+", "default": "-1",
"about": "The high watermark of the partition, or -1 if unknown." },
{ "name": "Batches", "type": "[]DisklessFindBatchesBatchInfo", "versions": "0+",
"about": "The batches found for this partition.", "fields": [
{ "name": "BatchId", "type": "int64", "versions": "0+",
"about": "The batch ID." },
{ "name": "ObjectKey", "type": "string", "versions": "0+",
"about": "The object key in object storage containing this batch." },
{ "name": "Magic", "type": "int8", "versions": "0+",
"about": "The record batch magic byte." },
{ "name": "ByteOffset", "type": "int64", "versions": "0+",
"about": "The byte offset of the batch within the object." },
{ "name": "ByteSize", "type": "int64", "versions": "0+",
"about": "The size of the batch in bytes." },
{ "name": "BaseOffset", "type": "int64", "versions": "0+",
"about": "The base offset of the record batch." },
{ "name": "LastOffset", "type": "int64", "versions": "0+",
"about": "The last offset of the record batch." },
{ "name": "LogAppendTimestamp", "type": "int64", "versions": "0+",
"about": "The log append timestamp." },
{ "name": "BatchMaxTimestamp", "type": "int64", "versions": "0+",
"about": "The maximum timestamp in the batch." },
{ "name": "TimestampType", "type": "int8", "versions": "0+",
"about": "The timestamp type (0 = CreateTime, 1 = LogAppendTime)." }
]}
]}
]}
]
} |
DisklessDescribeFiles
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"apiKey": 100,
"type": "request",
"listeners": ["controller"],
"name": "DisklessDescribeFilesRequest",
// Version 0 is the initial version.
"validVersions": "0",
"flexibleVersions": "0+",
"fields": [
{ "name": "Files", "type": "[]DisklessDescribeFileEntry", "versions": "0+",
"about": "Each file to describe.", "fields": [
{ "name": "ObjectKey", "type": "string", "versions": "0+", "mapKey": true,
"about": "The object key identifying the file in object storage." }
]},
{ "name": "TimeoutMs", "type": "int32", "versions": "0+", "default": "60000",
"about": "How long to wait in milliseconds before timing out the request." }
]
}
{
"apiKey": 100,
"type": "response",
"name": "DisklessDescribeFilesResponse",
// Version 0 is the initial version.
"validVersions": "0",
"flexibleVersions": "0+",
"fields": [
{ "name": "ThrottleTimeMs", "type": "int32", "versions": "0+", "ignorable": true,
"about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
{ "name": "Files", "type": "[]DisklessDescribeFileResult", "versions": "0+",
"about": "Results for each file.", "fields": [
{ "name": "ObjectKey", "type": "string", "versions": "0+", "mapKey": true,
"about": "The object key identifying the file in object storage." },
{ "name": "ErrorCode", "type": "int16", "versions": "0+",
"about": "The error code, or 0 if there was no error." },
{ "name": "ErrorMessage", "type": "string", "versions": "0+", "nullableVersions": "0+", "ignorable": true, "default": "null",
"about": "The error message, or null if there was no error." },
{ "name": "HasLiveBatches", "type": "bool", "versions": "0+",
"about": "True if the file still has live batches tracked by this coordinator." }
]}
]
} |
Metadata records
DisklessPartitionCreatedRecord
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"apiKey": 100,
"type": "data",
"name": "DisklessPartitionCreatedRecord",
// Version 0 is the initial version.
"validVersions": "0",
"flexibleVersions": "0+",
"fields": [
{ "name": "TopicId", "type": "uuid", "versions": "0+",
"about": "The unique topic ID." },
{ "name": "Name", "type": "string", "versions": "0+", "entityType": "topicName",
"about": "The topic name." },
{ "name": "PartitionIndex", "type": "int32", "versions": "0+",
"about": "The partition index." }
]
} |
DisklessTopicDeletedRecord
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"apiKey": 101,
"type": "data",
"name": "DisklessTopicDeletedRecord",
// Version 0 is the initial version.
"validVersions": "0",
"flexibleVersions": "0+",
"fields": [
{ "name": "TopicId", "type": "uuid", "versions": "0+",
"about": "The unique topic ID." }
]
}
|
DisklessFileCommittedRecord
TODO: optimize for storing fewer bytes per record
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"apiKey": 102,
"type": "data",
"name": "DisklessFileCommittedRecord",
// Version 0 is the initial version.
"validVersions": "0",
"flexibleVersions": "0+",
"fields": [
{ "name": "ObjectKey", "type": "string", "versions": "0+",
"about": "The object key identifying the file in object storage." },
{ "name": "ObjectFormat", "type": "int8", "versions": "0+",
"about": "The format of the object (e.g. 1 = WRITE_AHEAD_MULTI_SEGMENT)." },
{ "name": "FileSize", "type": "int64", "versions": "0+",
"about": "The total size of the file in bytes." },
{ "name": "UploaderBrokerId", "type": "int32", "versions": "0+", "entityType": "brokerId",
"about": "The broker that uploaded the file." },
{ "name": "OwnerIds", "type": "[]int32", "versions": "0+", "entityType": "brokerId",
"about": "The list of Diskless Coordinator IDs that share ownership of this file." },
{ "name": "Topics", "type": "[]DisklessCommittedTopic", "versions": "0+",
"about": "Each topic with accepted batches in this file.", "fields": [
{ "name": "TopicId", "type": "uuid", "versions": "0+",
"about": "The unique topic ID." },
{ "name": "Partitions", "type": "[]DisklessCommittedPartition", "versions": "0+",
"about": "Each partition with accepted batches for this topic.", "fields": [
{ "name": "PartitionIndex", "type": "int32", "versions": "0+",
"about": "The partition index." },
{ "name": "Batches", "type": "[]DisklessCommittedBatch", "versions": "0+",
"about": "The accepted batches for this partition.", "fields": [
{ "name": "Magic", "type": "int8", "versions": "0+",
"about": "The record batch magic byte." },
{ "name": "ByteOffset", "type": "int32", "versions": "0+",
"about": "The byte offset of this batch within the file." },
{ "name": "ByteSize", "type": "int32", "versions": "0+",
"about": "The size of the batch in bytes." },
{ "name": "AssignedBaseOffset", "type": "int64", "versions": "0+",
"about": "The base offset assigned by the coordinator." },
{ "name": "AssignedLastOffset", "type": "int64", "versions": "0+",
"about": "The last offset assigned by the coordinator." },
{ "name": "LogAppendTimestamp", "type": "int64", "versions": "0+",
"about": "The log append timestamp assigned by the coordinator." },
{ "name": "BatchMaxTimestamp", "type": "int64", "versions": "0+",
"about": "The maximum timestamp in the batch." },
{ "name": "TimestampType", "type": "int8", "versions": "0+",
"about": "The timestamp type (0 = CreateTime, 1 = LogAppendTime)." },
{ "name": "ProducerId", "type": "int64", "versions": "0+",
"about": "The producer ID, or -1 if not set." },
{ "name": "ProducerEpoch", "type": "int16", "versions": "0+",
"about": "The producer epoch, or -1 if not set." },
{ "name": "BaseSequence", "type": "int32", "versions": "0+",
"about": "The base sequence number, or -1 if not set." },
{ "name": "LastSequence", "type": "int32", "versions": "0+",
"about": "The last sequence number, or -1 if not set." }
]}
]}
]}
]
} |
DisklessRecordsDeletedRecord
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"apiKey": 103,
"type": "data",
"name": "DisklessRecordsDeletedRecord",
// Version 0 is the initial version.
"validVersions": "0",
"flexibleVersions": "0+",
"fields": [
{ "name": "TopicId", "type": "uuid", "versions": "0+",
"about": "The unique topic ID." },
{ "name": "PartitionIndex", "type": "int32", "versions": "0+",
"about": "The partition index." },
{ "name": "NewLogStartOffset", "type": "int64", "versions": "0+",
"about": "The new log start offset after deletion. All batches with last offset less than this value are removed." }
]
} |
Configuration
Broker configuration
More to be added as discussion unrolls
| Configuration | Description | Values |
|---|---|---|
diskless.metadata.topic.num.partitions | The number of Diskless metadata partitions and the number of Diskless coordinators. | |
diskless.coordinator.orphan.scan.interval.ms | The interval for scanning for orphan files. | |
diskless.coordinator.orphan.scan.grace.period.ms | The grace period to give to deletable files. |
Monitoring
More to be added as discussion unrollsTBD
Compatibility, Deprecation, and Migration Plan
...