Versions Compared

Old Version 13

changes.mady.by.user Lucas Bradstreet

Saved on

compared with

New Version 14

changes.mady.by.user Lucas Bradstreet

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Fetch

To avoid issues where FetchRequest(s) requests are made to stale topicspartitions, a topic_id field will be added to the request and responsefence reads for deleted topics.

FetchRequest v12

Fetch Request (Version: 12) => replica_id max_wait_time min_bytes max_bytes isolation_level session_id session_epoch [topics] [forgotten_topics_data] rack_id
  replica_id => INT32
  max_wait_time => INT32
  min_bytes => INT32
  max_bytes => INT32
  isolation_level => INT8
  session_id => INT32
  session_epoch => INT32
  topics => topic topic_id* [partitions]
    topic => STRING
    topic_id* => UUID
    partitions => partition current_leader_epoch fetch_offset log_start_offset partition_max_bytes
      partition => INT32
      current_leader_epoch => INT32
      fetch_offset => INT64
      log_start_offset => INT64
      partition_max_bytes => INT32
  forgotten_topics_data => topic [partitions]
    topic => STRING
    partitions => INT32
  rack_id => STRING

...

ListOffsets

To avoid issues where ListOffsetsRequest(s) requests are made to stale topicspartitions, a topic_id field will be added to the request and responsefence reads for deleted topics.

ListOffsetsRequest v6

ListOffsets Request (Version: 6) => replica_id isolation_level [topics]
  replica_id => INT32
  isolation_level => INT8
  topics => topic topic_id* [partitions]
    topic => STRING
    topic_id* => UUID
    partitions => partition current_leader_epoch timestamp
      partition => INT32
      current_leader_epoch => INT32
      timestamp => INT64

...

OffsetForLeader

To avoid issues where OffsetForLeaderRequest(s) requests are made to stale topicspartitions, a topic_id field will be added to the request and responsefence reads for deleted topics.

OffsetForLeaderRequest v4

...

It would be ideal if the log.dir layout could be restructured from {topic}_{partition} format to {{topicIdprefix}}/{topicId}_{partition}, e.g. "mytopic_1" → "24/24cc4332-f7de-45a3-b24e-33d61aa0d16c_1". Note the hierarchical directory structure using the first two characters of the topic ID to avoid having too many directories at the top level of the logdir.

A broker cannot migrate partitions to the new directory structure until the inter.broker.protocol.version is bumped, as this will not be a backwards compatible change.

The migration may be tricky in other ways. The broker opens each partition on broker startup, before the LeaderAndIsrRequest is received. A broker will only know topic IDs for all of its TopicPartitions after receiving the first V3 LeaderAndIsrRequest for that partition. Moving directories around will be difficult at this point, as the logdirs are open, the broker needs to respond to the LeaderAndIsrRequest, etc.

Maybe the broker should set the topic ID in the metadata file first, and then move directories on the next startup before each log is openedThis change is not required for the topic deletion improvements above, and will be left for a future KIP where it may be required e.g. topic renames.