THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
StatusStatus
| Page properties |
|---|
...
Document the state by adding a label to the FLIP page with one of "discussion", "accepted", "released", "rejected".
|
...
|
...
|
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
We provided a benchmark [1]. It not only increases the size of data sent over the network but also hurts performance as seen in the preliminary results below. In this results cloud-11 has 25 nodes and ibm-power has 8 nodes with scaling the number of slots per node from 1 - 16.
...
suite | name | median_time |
broadcast.cloud-11 | broadcast.01 | 8796 |
broadcast.cloud-11 | broadcast.02 | 14802 |
broadcast.cloud-11 | broadcast.04 | 30173 |
broadcast.cloud-11 | broadcast.08 | 56936 |
broadcast.cloud-11 | broadcast.16 | 117507 |
broadcast.ibm-power-1 | broadcast.01 | 6807 |
broadcast.ibm-power-1 | broadcast.02 | 8443 |
broadcast.ibm-power-1 | broadcast.04 | 11823 |
broadcast.ibm-power-1 | broadcast.08 | 21655 |
broadcast.ibm-power-1 | broadcast.16 | 37426 |
After looking into the code base it, it seems that the data is deserialized only once per TM, but the actual data is sent for all slots running the operator with broadcast vars and just gets discarded in case it’s already deserialized.
...
Fifth Approach - Write broadcast data to one blocking subpartition only
This approach writes the broadcast data only to one blocking subpartition. So there is no data redundancy and no unnecessary network traffic. In order to make the broadcast subpartitions blocking we modify the job graph generation accordingly.
...
In the case of a simple Map().withBroadcastSet() we achieve the following results on the IBM-cluster:
...
suite | name | average_time (ms) |
broadcast.ibm-power-1 | broadcast.01 | 6597.33 |
broadcast.ibm-power-1 | broadcast.02 | 5997 |
broadcast.ibm-power-1 | broadcast.04 | 6576.66 |
broadcast.ibm-power-1 | broadcast.08 | 7024.33 |
broadcast.ibm-power-1 | broadcast.16 | 6933.33 |
You can see that the run time stays constant, which is exactly what we want. So the next step is to extend iterations to make it work in general.
...
In the case of a simple Map().withBroadcastSet() we achieve the following results on the IBM-cluster:
suite | name | average_time (ms) |
broadcast.ibm-power-1 | broadcast.01 | 6433 |
broadcast.ibm-power-1 | broadcast.02 | 6162.66 |
broadcast.ibm-power-1 | broadcast.04 | 6459 |
broadcast.ibm-power-1 | broadcast.08 | 6635.66 |
broadcast.ibm-power-1 | broadcast.16 | 7002.66 |
Compatibility, Deprecation, and Migration Plan
...