...

Alternative Design - Adam Bellemare July 2018

Features:

1. Performs updates from both sides of the join.
2. Uses user-specified serdes to perform the prefixScans.
3. Resolves out-of-order processing due to foreignKey changes.
4. Fully scalable with increasing data, provided that one thread doesn't process most of the foreign-key data.

Design Philosophy

1. The foreignKeyJoin implementation doesn't require the user to have any knowledge about the inner workings of the join.
2. All events are returned on the same key as they were provided (K,V in, K,V0 out). This is the same as the regular join function.
3. Some Serdes are required to be provided by the user, as they are used to handle the CombinedKey serialization / deserialization.
4. The user can provide a custom partitioner if they so wish. This allows them to not have to rely strictly on the DefaultPartitioner if they have custom partitioning logic.

...