Page History

...

Our goal is to update the aggregate view of tables A, B, and C in real time.

Traditional Lookup Join Limitations

The traditional lookup join approach is limited by its sole response to changes in the main data stream. This means that if a related dimension table (such as B or C) undergoes changes, the data that has already been joined cannot be dynamically updated. In other words, a lookup operation is only triggered when a new event enters the main stream, and updates in dimension tables do not result in changes to the join results.

New Strategy: Dynamic Dimension Table-Driven Lookup Join

To overcome this limitation, we have adopted a new strategy where all tables (A, B, and C) are treated as dynamic dimension tables. We no longer regard any table as static; all changes are dynamically reflected in the main table. The core idea is to use changes in dimension tables to trigger data enrichment.

For example, when table C changes, we first find the related content in table B using the changed data's n field, and then retrieve the value of the m field. This m value becomes our new primary key, based on which we perform the lookup join operation, updating the aggregate view of A, B, and C in real time.

...