Table of Contents

Status

Current state: "Under Discussion"

Discussion thread:

JIRA: 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

In the cloud era, shared storage architecture is increasingly popular. Almost all cloud providers offer three types of shared storage services: block storage, file storage, and object storage. These services have many advantages over traditional local disks:

...

A stream is truly append-only data streaming with extremely simple APIs. Similar to how a log is divided into LogSegments, a stream consists of multiple StreamSlices, each being a short section of the stream.

public interface Stream {
    long streamId();

    long streamEpoch();

    long startOffset();

    long confirmOffset();

    long nextOffset();

    CompletableFuture<AppendResult> append(AppendContext context, RecordBatch recordBatch);

    CompletableFuture<FetchResult> fetch(FetchContext context, long startOffset, long endOffset, int maxBytesHint);

    CompletableFuture<Void> trim(long newStartOffset);

    CompletableFuture<Void> close();
    
    CompletableFuture<Void> destroy();
}

...

Each partition now has a shared log with multiple streams, necessitating storage for relationship information to identify the meta stream's Stream ID for a specific partition. This KIP recommends using KRaft to store this data, requiring the addition of three KRaft controller APIs and a KV client implemented by KRaft.

public interface KVClient {
    CompletableFuture<Value> putKVIfAbsent(KeyValue keyValue);

    CompletableFuture<Value> putKV(KeyValue keyValue);

    CompletableFuture<Value> getKV(Key key);

    CompletableFuture<Value> delKV(Key key);
}

GET_KVS(ApiMessageType.GET_KVS, false, true),
PUT_KVS(ApiMessageType.PUT_KVS, false, true),
DELETE_KVS(ApiMessageType.DELETE_KVS, false, true),

...