THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Close the gap between the semantics of KTables in streams and tables in relational databases. It is common practice to capture changes as they are made to tables in a RDBMS into kafka Kafka topics (JDBC-connect, Debezium, Maxwell). These entities typically have multiple one-to-many relationship. Usually RDBMSs offer good support to resolve this relationship with a join. Streams falls short here and the workaround (group by - join - lateral view) is not well supported as well and is not in line with the idea of record based processing.
...
The overall process is fairly simple and is outlined below.
Gliffy Diagram name simplifiedOverviewJohnWorkflow pagePin 31
More Detailed Implementation Details
...
- There is no impact to existing users.
Rejected Alternatives:
Shipping the whole payload across and having the Foreign-node perform the join.
Gliffy Diagram name simplifiedOverview pagePin 3
Rejected Because:
- Transfers the maximum amount of data across a network
- Joining on the foreign node can be a bottle-neck vs. joining on the local node.
Problem: Out-of-order processing of Rekeyed data
Solution A - Hold Ordering Metadata in Record Headers and Highwater Mark Table
...