Current state: Under Discussion
Discussion thread: here
JIRA: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
The log.segment.bytes broker config (and its topic-level synonym segment.bytes) is currently defined as ConfigDef.Type.INT, capping the maximum segment size at Integer.MAX_VALUE (2,147,483,647 bytes, approximately 2 GB). Additionally, the .index file format stores physical file positions as 4-byte signed integers, which also cannot address beyond approximately 2 GB.
With modern storage hardware (multi-TB NVMe drives) and high-throughput workloads, the 2 GB cap is increasingly a problem:
.log, .index, .timeindex, .txnindex). A 10 TB partition with 2 GB segments means approximately 20,000 open files.Allowing segments of 4 GB, 8 GB, or larger would significantly reduce these overheads for high-throughput, large-retention workloads.
| Config | Current type | New type | Current range | New range |
|---|---|---|---|---|
log.segment.bytes (broker) | INT | LONG | [1 MB, 2,147,483,647] | [1 MB, Long.MAX_VALUE] (after MetadataVersion finalization) |
segment.bytes (topic) | INT | LONG | [1 MB, 2,147,483,647] | [1 MB, Long.MAX_VALUE] (after MetadataVersion finalization) |
The expanded range (values greater than Integer.MAX_VALUE) is gated by MetadataVersion. Before finalization, the effective range remains [1 MB, Integer.MAX_VALUE]. After IBP_4_4_IV1 is finalized, the range becomes [1 MB, Long.MAX_VALUE].
Offset index (.index) file format -- new 12-byte entry format (gated by MetadataVersion):
| Field | Legacy format (8 bytes per entry) | Large format (12 bytes per entry) |
|---|---|---|
| Relative offset | 4-byte signed int | 4-byte signed int |
| Physical position | 4-byte signed int (max approximately 2 GB) | 8-byte signed long (effectively unlimited) |
The large format is only written after MetadataVersion IBP_4_4_IV1 is finalized. Before finalization, all index files use the legacy 8-byte format.
Time index (.timeindex) -- no format change. Entry size is already 12 bytes (8-byte timestamp + 4-byte relative offset). No physical positions are stored.
Transaction index (.txnindex) -- no format change. Uses FileChannel directly with long positions.
| Class | Member | Before | After |
|---|---|---|---|
LogConfig | DEFAULT_SEGMENT_BYTES | int | long |
LogConfig | segmentSize() | returns int | returns long |
LogConfig | initFileSize() | returns int | returns long |
AbstractKafkaConfig | logSegmentBytes() | returns Integer via getInt() | returns Long via getLong() |
RollParams | maxSegmentBytes | int | long |
OffsetIndex | append(long offset, ...) | int position | long position |
OffsetPosition | position field | int | long |
FileRecords | internal size field | AtomicInteger | AtomicLong |
LogSegment | new sizeInBytesLong() | N/A | returns long |
LogSegment | recover() | returns int | returns long |
LogSegment | truncateTo() | returns int | returns long |
LogOffsetMetadata | relativePositionInSegment | int | long |
SegmentPosition (raft) | relativePosition | int | long |
RemoteStorageManager | fetchLogSegment(metadata, int) | only overload | @Deprecated; new default method with long added |
RemoteStorageManager | fetchLogSegment(metadata, int, int) | only overload | @Deprecated; new default method with long, long added |
BaseRecords.sizeInBytes() remains int (449 callers across 89 files -- cascading this change is too large for this KIP).RecordBatch.sizeInBytes() remains int (bounded by max.message.bytes).MemoryRecords.sizeInBytes() remains int (bounded by ByteBuffer capacity).transaction.state.log.segment.bytes, offsets.topic.segment.bytes, share.coordinator.state.topic.segment.bytes) remain INT.No new metrics are added. Existing segment size metrics will report accurate values for segments larger than 2 GB because LogSegments.sizeInBytes() uses long arithmetic internally.
kafka-log-dirs.sh and DumpLogSegments correctly handle segments larger than 2 GB. DumpLogSegments uses the sliceLong() method for position-based slicing.
Change log.segment.bytes and segment.bytes from ConfigDef.Type.INT to ConfigDef.Type.LONG. Apply atLeast(1024 * 1024) as the validator. The expanded range (values greater than Integer.MAX_VALUE) is gated by MetadataVersion IBP_4_4_IV1.
Widen internal storage layer types from int to long for segment sizes and physical file positions:
FileRecords: Internal AtomicInteger changed to AtomicLong for size tracking. New sizeInBytesLong(), sliceLong(), truncateToLong() methods added for callers that need long precision. The existing BaseRecords.sizeInBytes() interface remains int to avoid cascading changes.
LogSegment: New sizeInBytesLong() alongside existing size(). Methods recover(), append(), shouldRoll(), read(), and truncateTo() widened to use long for positions and sizes.
OffsetIndex dual format: The OffsetIndex supports two entry formats, controlled by a useLargeFormat constructor parameter:
The default is legacy format (useLargeFormat=false). All production code paths (via LazyIndex, RemoteIndexCache, DumpLogSegments) create OffsetIndex instances in legacy format. The large format is only used when explicitly opted in via the 5-arg constructor after MetadataVersion verification.
The AbstractIndex base class is enhanced with an effectiveEntrySize() method that safely resolves the entry size at construction time without calling overridable methods from the constructor.
MetadataVersion gating: A new MetadataVersion entry IBP_4_4_IV1(32, "4.4", "IV1", true) gates the format change. The didMetadataChange=true flag ensures downgrade is blocked after finalization. The helper method isLargeIndexFormatSupported() returns true when the cluster MetadataVersion is at or above IBP_4_4_IV1.
RemoteStorageManager API: New default methods added to the RemoteStorageManager interface with long position parameters. These delegate to the existing int methods with bounds checking. Existing RemoteStorageManager implementations continue to work unchanged. The old int methods are marked @Deprecated.
Cascading type changes: Other code that consumes sizeInBytes() or OffsetPosition.position is updated. Key call sites include UnifiedLog, Cleaner, LogLoader, LocalLog, RemoteLogManager, RemoteIndexCache, DelayedFetch, and DumpLogSegments.
kafka-features.sh upgrade --release-version 4.4. New index files are written in 12-byte format. Existing 8-byte index files continue to be read correctly. When an index is rebuilt (for example during LogSegment.recover()), it is written in the new format. The segment.bytes config upper bound is lifted.IBP_4_4_IV1 has didMetadataChange=true, consistent with existing KRaft downgrade rules.INT values stored as strings (for example "1073741824") parse correctly as LONG. No user action required.int methods. The new long default methods delegate to the old int methods with bounds checking. No changes required for existing RSM plugins.segment.bytes back to 2 GB or less and wait for segment rolls before downgrading.RemoteStorageManager.fetchLogSegment(RemoteLogSegmentMetadata, int) is deprecated in favor of fetchLogSegment(RemoteLogSegmentMetadata, long).RemoteStorageManager.fetchLogSegment(RemoteLogSegmentMetadata, int, int) is deprecated in favor of fetchLogSegment(RemoteLogSegmentMetadata, long, long).segment.bytes above the current default (1 GB) are completely unaffected.MetadataVersion to IBP_4_4_IV1.RemoteStorageManager implementations should migrate to the long overloads at their convenience. The deprecated int methods continue to work.LONG type works correctly for log.segment.bytes and segment.bytes.Integer.MAX_VALUE.sizeInBytesLong() returns accurate values exceeding Integer.MAX_VALUE, and truncateToLong() handles truncation amounts exceeding Integer.MAX_VALUE.shouldRoll() works with maxSegmentBytes greater than 2 GB, sizeInBytesLong() consistency with size() for small segments, and recovery preserves all records.long-param fetchLogSegment() methods work correctly.Rejected. The 2 GB limit is an artificial constraint from a type choice made when storage hardware was smaller. Modern deployments routinely manage multi-TB partitions where 2 GB segments create excessive overhead in file handles, segment rolls, compaction cycles, and tiered storage operations.
Rejected. Maintaining two configs for the same purpose adds confusion for operators. A single config with a type change is cleaner and follows the precedent set by KIP-1161, which reclassified several configs from STRING to LIST type.
Rejected. The relative offset (4 bytes) is sufficient because it represents the delta from the segment base offset, not an absolute offset. Only the physical position needs widening to 8 bytes. Using 16 bytes per entry would waste 50% more space for no practical benefit.
Rejected. Java does not natively support unsigned integers, making the code error-prone (values above Integer.MAX_VALUE appear negative, breaking comparison operators and binary search). The additional 2 GB headroom is not worth the complexity. Widening to long is the clean solution and future-proofs the format.
Rejected as the complete approach. While Phase 1 (config type change with INT range cap) is useful as a stepping stone, it does not deliver the actual user-facing value of larger segments. A single KIP covering the full scope ensures the community reviews the complete design, even though implementation can be phased across multiple PRs.
Rejected. An earlier prototype used a .index_version marker file in each partition directory to track whether indexes had been rebuilt in the new format. On first startup after upgrade, if the marker was absent, all indexes were rebuilt. This approach was rejected because:
Rejected. Adding a version byte at the start of each index file would allow self-describing format detection, but it adds complexity to the read path (must check header before every index open), and old brokers would misread the header byte as part of the first entry, potentially producing garbled offset lookups before sanity checks catch it. MetadataVersion gating is simpler and avoids these edge cases entirely.