...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Reduce the gap between the semantics of  relational databases and data streams.

It is a common

...

data integration technique to capture RDMS database changes as they are made to data entities

...

using Change Data Capture platform

...

and send these changes as messages to Kafka topics.

Often each data entity is stored in several tables with one-to-many relationships between

...

them which allows for the easy

...

extraction of these entities from the DB using SQL SELECT statements with foreign key table join.

Also reading the data entities from the application database usually associated with some specific "business" event when data entity is in the "business" consistent state. For example Order data entity on the diagram below could only be meaningfully extracted for the integration when its state is "COMPLETED" since before that it might be still in the process of creation.

Streams falls short here and the workaround (group by - join - lateral view) is not well supported as well and is not in line with the idea of record based processing.

This KIP makes relational data liberated by connection mechanisms far easier for teams to use, smoothing a transition to natively-built event-driven services.

Public Interfaces

Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.

...