Page History

...

Another popular pattern for enforcing platform standards and distributing features to all clients is through the deployment of what I refer to as (special-purpose) platform apps. Think of the stream processing application that performs some sort of stateless transformation on every topic message, and writes the result back to Kafka. In many instances, this architecture might be warranted (e.g. if the computations are expensive for instanceor the logic is complex). But in a lot of cases, these systems emerged in response to as a way to work around the library extension pattern’s limitations since it allows and allow platform owners to save everyone the headache deploy features at scale without managing the lifecycle of library upgradesextensions. Unfortunately, using platform apps to enforce standards is an expensive strategy with rather poor scalability features. For instance, if a company decides to solve privacy enforcement through a platform app that redacts every Kafka message in-flight, the system will double disk space, the number of partitions and require a large amount of additional compute.

...

This KIP wants to provide a third way for platform owners: the ability to extend Kafka’s server-side behavior through broker interceptors. At their core, broker interceptors can be thought of as very lightweight, stateless stream processors that can intercept, mutate and mutate filter messages either at produce or consume time.

...

To date, Kafka hasn't released a comparable feature to OSS users (though Confluent server does support server-side schema validation). This KIP is proposing , and this KIP wants to to change that through the addition of broker interceptors to the stack.

...