Page History

...

2. Solves the data skew problem after introduces hash lookup join

3. As we know, in Hive dimension source, each task would load all data into cache. After introduce hash partitioner in point 1, each task could only load part of cache instead of load all cache.

4. Enables mini-batch optimization to reduce RPC call

We would focus on point 1 in this FLIP, and continue to discuss point2, point3 and point 3 4 in the later FLIP.

Many Lookup table sources introduce cache to reduce the RPC call, such as JDBC, CSV, HBase connectors.

For those connectors, we could raise cache hit ratio by routing the same lookup keys to the same task instance. This is the purpose of this FLIP.

There are many similar requirements from user mail list and JIRA about hash Lookup Join, for example:

...