THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
An new option to control minibatch join is added.
Key | Default | Type |
table.exec.stream.join.mini-batch-enabled | false | boolean |
In addition, other three options is required to set to make minibatch join work. The example is as following. Go to options for more details about other three options .
...
A new operator called MiniBatchStreamingJoinOperator which inherits from StreamingJoinOperator is introduced. It now supports four types of joins: left join, right join, full join and inner join (not support semi / anti join). It will use a block of memory as a minibatch to store data input from the left and right streams like HeapBufferedBundle for GroupAggregate operation. The records in the minibatch is processed when triggered by minibatch size or watermark.
It will use a block of memory as a minibatch to store data input from the left and right streams like HeapBufferedBundle for GroupAggregate operation. The records in the minibatch is processed when triggered by minibatch size or watermark. The following is the UML of MiniBatchStreamingJoinOperator.
The explaination of 3 points of optimization and its corresponding scenarios could be referenced in the appendix and and the nexmark metrics is also listed. Go to Appendix for more detailsBesides, go to implementation could get details of the POC.
Compatibility, Deprecation, and Migration Plan
...