Page History

...

The first step to addressing the above changes is to make the fetching of metadata asynchronous within the producer. This directly fixes (3), and opens the path for resolving (1) by enabling the metadata requests to be batched together. Since the producer's interface is asynchronous and it inherently batches the sending of records to partitions, subjecting the metadata fetching to a subset of the batching delay doesn't change the interaction or expectations of the client. This change alone should be good enough to bring performance back to acceptable (pending verification).

Specific modifications would be are to make KafkaProducer#waitOnMetadata to be asynchronous when it must block. For uncached topics, the producer will maintain a queue of its outstanding records to ensure proper ordering (in the accumulator and for callback invocations) once topic metadata is resolved. Proper care must be taken to maintain the linger period for fetching metadata, individual record timeouts while queued, and record buffer memory limits.

...