Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

More specifically, This operator will sort the input before aggregation, and avoid invoking window actions, which is similar to '5)'.

Analysis

After the proposed changes, the following DataStream API could specify that the window will end only at the end of its inputs when EndOfStreamWindows is used.

DataStream:

  • DataStream#cogroup

WindowedStream:

  • WindowedStream#reduce

  • WindowedStream#aggregate

  • WindowedStream#apply

  • WindowedStream#process

  • WindowedStream#sum

  • WindowedStream#min

  • WindowedStream#max

  • WindowedStream#minBy

  • WindowedStream#maxBy

KeyedStream:

  • KeyedStream#process

  • KeyedStream#intervalJoin

  • KeyedStream#reduce

  • KeyedStream#sum

  • KeyedStream#min

  • KeyedStream#max

  • KeyedStream#minBy

  • KeyedStream#maxBy

CoGroupedStream:

  • CoGroupedStream#WithWindow#apply

ConnectedStreams:

  • ConnectedStreams#process

Among of these API, when EndOfStreamWindows is used, WindowedStream#aggregate and DataStream#coGroup could get better performance because of the related optimization after this FLIP.

Benchmark results

To demonstrate our optimization improvements,we run each benchmark in different execution modes and configurations. Each result is measured by taking the average execution time across 5 runs with the given configuration.

...