Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Sliding window aggregation (Note: grace will be required in SlidingWindows instead of implementing the 24 hour default seen in TimeWindows and SessionWindows):

Code Block
stream.groupByKey().windowedBy(SlidingWindows.of(twentySeconds).grace(fiftySeconds).toStream()

...

Records that come out of order will be processed the same way as in-order records, as long as they fall within the grace period. As noted above, the grace period is required to create the windows. Two new windows will be created by the late record, one ending at that record's timestamp, and one starting at 1ms past that record's timestamp. These windows will be updated with existing records that fall into them by using aggregations previously calculated. Additionally, any existing windows that the new record falls within will be updated with a new aggregation that includes this late record. 

...

  • Allow users to specify granularity of window definition (something other than 1ms). This is an option for the future, but allowing a user to specify 1 second or 1 minute granularity results in windows that are very similar to tumbling windows, straying from the purpose of sliding windows.
  • Include a 'subtraction' feature as a complement to aggregation. While this might be a nice to have feature in the future, it's out of scope for this KIP and isn't necessary to make sliding windows work well.

Other Implementation Options:

  • Making SlidingWindows an option within TimeWindow. This would add more requirements for users and blur the lines between hopping and sliding windows, while potentially making optimization harder.
  • Making SlidingWindows a stand-alone class similar to SessionWindows. Because of the similarity in SlidingWindows and TimeWindows, leveraging the current TimeWindows implementation is preferable to creating all new processors that use very similar or the same code.