This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: Draft

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: here [Change the link from KAFKA-1 to your own ticket]

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Reduce the gap between the semantics of  relational databases and data streams.

It is a common data integration technique to capture RDMS database changes as they are made to data entities using Change Data Capture (CDC) platform and send these changes as messages to Kafka topics.

Often each data entity is stored in several tables with one-to-many relationships between them which allows for the easy extraction of these entities from the DB using SQL SELECT statements with foreign key table join.

Also reading the data entities from the application database usually associated with some specific "business" event when data entity is in the "business" consistent state. For example Order data entity on the diagram below could only be meaningfully extracted for the integration when its state is "COMPLETED" since before that it might be still in the process of creation.

In order to integrate RDBMS data entity with consumers using RDBMS->CDC->Kafka->KStreams->Consumer pipeline the following sequence of steps could be used:

• Replicate each entity table to its own topic with CDC. Typically this is only possible using Table PK as Kafka Message Key. We will call these topics "Data Topics" here.
• Create Kafka Stream KTables from all "Data Topics"
• Publish business events that trigger data entity extraction from Kafka topics to its own topic. We can call this topic "Trigger Topic" here
• Use existing stream/table join feature in Kafka Streams that will join "Trigger Topic" based stream with parent table of data entity (Orders) in our example
• Use new Kafka Streams feature proposed in this KIP-955 to join the stream resulting from the previous step with the rests of KTables using left Foreign Key Join

This KIP makes data aggregation semantic consistent between SQL and Kafka Streams.

Public Interfaces

Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.

A public interface is any change to the following:

• Binary log format

• The network protocol and api behavior

• Any class in the public packages under clientsConfiguration, especially client configuration

  • org/apache/kafka/common/serialization

  • org/apache/kafka/common

  • org/apache/kafka/common/errors

  • org/apache/kafka/clients/producer

  • org/apache/kafka/clients/consumer (eventually, once stable)

• Monitoring

• Command line tools and arguments

• Anything else that will likely break existing users in some way when they upgrade

Proposed Changes

Describe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?
• If we are changing behavior how will we phase out the older behavior?
• If we need special migration tools, describe them here.
• When will we remove the existing behavior?

Test Plan

Describe in few sentences how the KIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.