Page History

...

...

...

1. Collect deleted topics:
   1. Old format: /admin/delete_topics pulling the topic state from /brokers/topics/[topic].
   2. New in-memory topic deletion states from received DeleteTopicsRequest(s) 
2. Remove deleted topics from replicas by sending StopReplicaRequest V3 for any topics which do not contain a topic ID, and V4 for any topics which do contain a topic IDV3 before the IBP bump, and V4 after the IBP bump.
3. Finalize successful deletes:
   1. For /admin/delete_topics deletes, we may need to respond to the TopicDeleteRequest. We can also delete the topic znode at /admin/delete_topics/[topic] and /brokers/topics/[topic].
   2. For deletes for topics with topic IDs, remove the topic from the in memory topic deletion state on the controller.
4. Any unsuccessful StopReplicaRequest(s) will be retried after retryMs, starting from 1) and will be maintained in memory.

...

• Metadata schema version (schema_version: int32)
• Topic ID (id: UUID)

This file can either will be plain text (key/value pairs) or JSON..

It will be easy to update the file to include more fields in the future.

...

LeaderAndIsrRequest(s) will only be sent by the controller once a topic ID has been successfully assigned to the topic. The migration process can take place without an inter-broker protocol bump, as the format stored in /brokers/topics/[topic] will be compatible with older broker versions. 

When a replica receives a LeaderAndIsrRequest containing a topic ID for an existing partition which does not have a topic ID associated, it will create a partition metadata file for the topic partition locally. At this point the local partition will have been migrated to support topic IDs.

...

Ideally, consumer offsets stored in the __consumer_offsets topic would be associated with the topic ID for which they were read. However, given the way the __consumer_offsets is compacted, this may be difficult to achieve in a forwards compatible way. This change will be left until topic IDs are implemented in the clients.

Group Metadata Manager

Another future improvement opportunity is to use topicId in GroupMetadataManager.offsetCommitKey in the offset_commit topic. This may save some space.

log.dir layout

It would be ideal if the log.dir layout could be restructured from {topic}_{partition} format to {{topicIdprefix}}/{topicId}_{partition}, e.g. "mytopic_1" → "24/24cc4332-f7de-45a3-b24e-33d61aa0d16c_1". Note the hierarchical directory structure using the first two characters of the topic ID to avoid having too many directories at the top level of the logdir. This change is not required for the topic deletion improvements above, and will be left for a future KIP where it may be required e.g. topic renames.

...