This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: Under Discussion

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

The log.segment.bytes broker config (and its topic-level synonym segment.bytes) is currently defined as ConfigDef.Type.INT, capping the maximum segment size at Integer.MAX_VALUE (2,147,483,647 bytes, ~2 GB). Additionally, the .index file format stores physical file positions as 4-byte signed integers, which also cannot address beyond ~2 GB.

With modern storage hardware (multi-TB NVMe drives) and high-throughput workloads, the 2 GB cap is increasingly a problem:

• Excessive file handle usage: Each segment needs 4 files (.log, .index, .timeindex, .txnindex). A 10 TB partition with 2 GB segments means ~20,000 open files.
• Frequent segment rolls: A topic ingesting 500 MB/s rolls a new segment every ~4 seconds, amplifying index build, flush, and cleaner overhead.
• More log cleaning / compaction work: More segments means more compaction cycles with more small groups.
• Remote storage overhead: Each segment is an individual unit for tiered storage copy/delete operations.

Allowing segments of 4 GB, 8 GB, or larger would significantly reduce these overheads for high-throughput, large-retention workloads.

Public Interfaces

Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.

A public interface is any change to the following:

• Binary log format

• The network protocol and api behavior

• Any class in the public packages under clientsConfiguration, especially client configuration

  • org/apache/kafka/common/serialization

  • org/apache/kafka/common

  • org/apache/kafka/common/errors

  • org/apache/kafka/clients/producer

  • org/apache/kafka/clients/consumer (eventually, once stable)

• Monitoring

• Command line tools and arguments

• Anything else that will likely break existing users in some way when they upgrade

Proposed Changes

Describe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.

Config type change

Change log.segment.bytes / segment.bytes from ConfigDef.Type.INT to ConfigDef.Type.LONG. The expanded range (> Integer.MAX_VALUE) is gated by MetadataVersion. Before finalization, the effective range remains [1 MB, Integer.MAX_VALUE]. After IBP_4_4_IV1 is finalized, the range becomes [1 MB, Long.MAX_VALUE].

Storage layer widening

Widen the internal storage layer types from int to long for segment sizes and physical file positions:

• FileRecords: Internal AtomicInteger -> AtomicLong for size tracking. New sizeInBytesLong(), sliceLong(), truncateToLong() methods for callers that need long precision. The existing BaseRecords.sizeInBytes() interface remains int to avoid cascading changes across 449 call sites.
• LogSegment: New sizeInBytesLong() alongside existing size(). recover(), append(), shouldRoll(), read(), truncateTo() all widened to use long for positions and sizes.
• OffsetPosition: position field widened from int to long.
• LogOffsetMetadata: relativePositionInSegment widened from int to long.
• RollParams: maxSegmentBytes widened from int to long.

OffsetIndex dual format

The OffsetIndex supports two entry formats, controlled by a useLargeFormat constructor parameter:

The default is legacy format (useLargeFormat=false). All production code paths (via LazyIndex, RemoteIndexCache, DumpLogSegments) create OffsetIndex instances in legacy format. The large format is only used when explicitly opted in via the 5-arg constructor after MetadataVersion verification.

The AbstractIndex base class is enhanced with an effectiveEntrySize() pattern that safely resolves the entry size at construction time without calling overridable methods from the constructor.

MetadataVersion gating

A new MetadataVersion entry gates the format change:

IBP_4_4_IV1(32, "4.4", "IV1", true)  // didMetadataChange=true

• isLargeIndexFormatSupported() helper method returns true when the cluster's MetadataVersion >= IBP_4_4_IV1.
• didMetadataChange=true ensures downgrade is blocked after finalization, consistent with existing KRaft downgrade rules.

RemoteStorageManager API

New default methods added to the RemoteStorageManager interface with long position parameters:

default InputStream fetchLogSegment(RemoteLogSegmentMetadata metadata, long startPosition)
default InputStream fetchLogSegment(RemoteLogSegmentMetadata metadata, long startPosition, long endPosition)

These delegate to the existing int methods with bounds checking. Existing RemoteStorageManager implementations continue to work unchanged. The old int methods are marked @Deprecated.

New or Changed Public Interfaces

Configuration changes

On-disk format changes

Offset index (.index) file format -- new 12-byte entry format (gated by MetadataVersion):

Time index (.timeindex) -- no format change. Entry size is already 12 bytes (8-byte timestamp + 4-byte relative offset).

Transaction index (.txnindex) -- no format change.

Java API changes

Not changed

• BaseRecords.sizeInBytes() remains int (449 callers across 89 files -- too large to cascade).
• RecordBatch.sizeInBytes() remains int (bounded by max.message.bytes).
• MemoryRecords.sizeInBytes() remains int (bounded by ByteBuffer capacity).
• Coordinator segment configs (transaction.state.log.segment.bytes, offsets.topic.segment.bytes) remain INT.

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?
• If we are changing behavior how will we phase out the older behavior?
• If we need special migration tools, describe them here.
• When will we remove the existing behavior?

Rolling upgrade path

1. Upgrade all brokers to the new version. Do not finalize the metadata version yet.
   • Brokers write index files in legacy 8-byte format (identical to old brokers).
   • Full backward and forward compatibility. Downgrade is safe.
2. Finalize the metadata version via kafka-features.sh upgrade --release-version 4.4.
   • New index files are written in 12-byte format.
   • Existing 8-byte index files continue to be read correctly (OffsetIndex supports dual format).
   • When an index is rebuilt (e.g., during LogSegment.recover()), it is written in the new format.
   • The segment.bytes config upper bound is lifted.
3. Downgrade after finalization is blocked (IBP_4_4_IV1 has didMetadataChange=true), consistent with existing KRaft downgrade rules.

Backward compatibility

• Config parsing: INT values stored as strings (e.g., "1073741824") parse correctly as LONG.
• Index format: Before MetadataVersion finalization, all indexes use the legacy 8-byte format. Old and new brokers produce identical index files.
• RemoteStorageManager: Existing implementations only implement int methods. The new long default methods delegate to the old int methods with bounds checking.

Forward compatibility

• Before finalization: full downgrade is safe. All index files are legacy format.
• After finalization: downgrade is blocked by KRaft metadata version rules.

Verified scenarios

Test Plan

Describe in few sentences how the KIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.

1. Keep as INT permanently

Rejected. The 2 GB limit is an artificial constraint from a type choice made when storage was smaller. Modern deployments routinely manage multi-TB partitions where 2 GB segments create excessive overhead.

2. Add a separate log.segment.bytes.long config

Rejected. Maintaining two configs for the same purpose adds confusion. A single config with a type change is cleaner and follows the precedent set by KIP-1161 (STRING to LIST type reclassification).

3. Widen index entries to 16 bytes (8-byte offset + 8-byte position)

Rejected. The relative offset (4 bytes) is sufficient -- it represents the delta from the segment base offset, not an absolute offset. Only the physical position needs widening to 8 bytes.

4. Use unsigned int for physical position (4 GB range)

Rejected. Java does not natively support unsigned integers, making the code error-prone. Widening to long is the clean solution and future-proofs the format.

5. Do only the config change without the index format change

Rejected as the complete approach. While Phase 1 (config type change with INT range cap) is useful as a stepping stone, it doesn't deliver the actual user-facing value of larger segments. A single KIP covering the full scope ensures the community reviews the complete design.

6. Per-partition marker file for index format migration

Rejected. An earlier prototype used a .index_version marker file in each partition directory to track whether indexes had been rebuilt in the new format. This approach was rejected because:

• Does not follow Kafka conventions. Every other on-disk format change in Kafka uses MetadataVersion gating.
• No downgrade support. The marker file approach immediately writes new-format indexes on upgrade, with no way to revert. MetadataVersion gating allows all brokers to be upgraded (still writing old format) before the format switch is finalized.
• Mixed-version cluster risk. In a rolling upgrade, broker A would immediately start writing 12-byte indexes while broker B (not yet upgraded) still expects 8-byte.
• False negatives in format detection. Old 8-byte index files whose size is divisible by both 8 and 12 (e.g., 72 bytes = 9 entries) cannot be reliably detected by file-size heuristics alone, leading to silent data corruption on ~33% of index files.

7. Magic byte header in index files

Rejected. Adding a version byte at the start of each index file would allow self-describing format detection, but:

• Adds complexity to the read path (must check header before every index open).
• Old brokers would misread the header byte as part of the first entry, potentially producing garbled offset lookups before sanityCheck catches it.
• MetadataVersion gating is simpler and avoids these edge cases entirely.