This enhancement proposes an improvement to the current behavior of Window Evictor, by providing more control on how the elements are to be evicted from the Window. Original Design Document of this proposal can be found here

Status

Current state: Under Discussion

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).This enhancement proposes an improvement to the current behavior of Window Evictor, by providing more control on how the elements are to be evicted from the Window. Origianal Design Document of this proposal can be found here

Motivation

Right now, the ability of Window Evictor is limited

...

• Changes to the Evictor interface :  addition two new methods evictBefore and evictAfter, removal of the existing evict method
• Corresponding changes to CountEvicot, DeltaEvictor
• Renaming of the existing TimeEvictor as ProcessingTimeEvictor(since the current behavior of TimeEvictor does not consider the EventTime of the records) and addition of new class EventTime evictorEventTimeEvictor.
• New overloaded method CountEvictor.of() that takes an additional parameter doEvictAfter, which decides whether to do eviction after the WindowFunction.
• Similar overloaded methods in DeltaEvictor, ProcessingTimeEvictor and EventTimeEvictor.

Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.

A public interface is any change to the following:

• DataStream and DataSet API, including classes related to that, such as StreamExecutionEnvironment

Proposed Changes

Proposed Changes

We propose to change the interface of Evictor to this

public interface Evictor<T, W extends Window> extends Serializable {

   /**
    * Optionally evicts elements. Called before windowing function.
    * @param elements The elements currently in the pane.
    * @param size The current number of elements in the pane.
    * @param window The {@link Window}
    * @param evictorContext The context for the Evictor
     */
   void evictBefore(Iterable<StreamRecord<T>> elements, int size, W window, EvictorContext evictorContext);

   /**
    * Optionally evicts elements. Called after windowing function.
    * @param elements The elements currently in the pane.
    * @param size The current number of elements in the pane.
    * @param window The {@link Window}
    * @param evictorContext The context for the Evictor
    */
   void evictAfter(Iterable<StreamRecord<T>> elements, int size, W window, EvictorContext evictorContext);
 
   /**
   * A context object that is given to {@link Evictor} methods
   */
   interface EvictorContext {}

The Evictor has two methods, evictBefore  - called before the WindowFunction - and evictAfter - called after the WindowFunction. These methods receive an Iterable of all the elements in the pane and the number of elements in the pane. Evictor can choose to remove elements from the Iterable based on any condition that a user wish to implement. The EvictorContext is an interface very similar to TriggerContext but for evictors, I’m omitting it for brevity.

This model allows to express everything that was possible with the current model but also 

• Allows eviction of elements from the “middle” of the buffer. 

• Allows eviction to be done before and/or after the WindowFunction.

CountEvictor, DeltaEvictor, ProcessingTimeEvictor and EventTimeEvictor by default will evict elements before the window function. 

DeltaEvictor will always iterate through the entire elements and evict if the element has higher delta than the threshold. EventTimeEvictor will iterate through all the elements and remove the elements with timestamp <= evictCutoff. ProcessingTimeEvictor will evict elements from the window till it finds the first element that has a timestamp <= the evictCutoff (Since we are not concerned about out-of-order events). CountEvictor will only iterate through the elements till the count of elements in the window is reduced to specified size. Describe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.

Compatibility, Deprecation, and Migration Plan

...