This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: Under Discussion

Discussion thread: here

JIRA: [Not present]


Motivation

Kafka usually is used in high-performance and low latency event processing systems to save mission critical information. In such systems event processing latency is critical and must met very strict requirements - e.g. 1 millisecond per event. Kafka producer in general meet that requirement. It is asynchronous and doesn't wait for any network operation in caller thread. Except for metadata update. Usually metadata update is done periodically with ProducerConfig#METADATA_MAX_AGE_CONFIG interval.

But if some topic is rarely used then its metadata can expire and sending new data for that topic can be blocked by waiting latest metadata.

In order to mitigate this problem and get rid of application thread blocking by metadata update in case of sending record for rarely updated topics I propose make Metadata#TOPIC_EXPIRY_MS configurable. And make TOPIC_EXPIRY_MS more than interval of sending events for rarely used topic.

Public Interfaces

New Metadata constructor must be introduced with topicExpiryMs parameter.

Proposed Changes

Need to introduce new client parameter TOPIC_EXPIRY_MS_CONFIG and pass its value to Metadata constructor during Metadata object creation.

Compatibility, Deprecation, and Migration Plan

  • Changes are 100% backward compatible.
  • We can mark current Metadata constructor without topicExpiryMs parameter as deprecated and make sure it is not used in kafka code base. And then remove it according to kafka deprecation policy.

Rejected Alternatives

Another approach to leverage main application thread block during sending events to kafka is to create separate queue for records that wait metadata. That is more expensive than making Metadata#TOPIC_EXPIRY_MS configurable.


  • No labels