Status

Current state: Under Discussion

Discussion thread: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enhance-Window-Evictor-in-Flink-td12406.html

JIRA: https://issues.apache.org/jira/browse/FLINK-4174

Released: 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

This enhancement proposes an improvement to the current behavior of Window Evictor, by providing more control on how the elements are to be evicted from the Window. Origianal Design Document of this proposal can be found here

Motivation

Right now, the ability of Window Evictor is limited

• The Evictor is called only before the WindowFunction. (There can be use cases where the elements have to be evicted after the WindowFunction is applied)
• Elements are evicted only from the beginning of the Window. (There can be cases where we need to allow eviction of elements from anywhere within in the Window as per the eviction logic that user wish to implement)

The current interface of Evictor is this:

public interface Evictor<T, W extends Window> extends Serializable {
/**
* Computes how many elements should be removed from the pane. The result specifies how
* many elements should be removed from the beginning. *
* @param elements The elements currently in the pane.
* @param size The current number of elements in the pane.
* @param window The {@link Window} */
int evict(Iterable<StreamRecord<T>> elements, int size, W window); 
}

The Evictor gets an Iterable for all the window elements and the number of elements in the window. Then it can return a number that specifies how many elements should be evicted, starting from the beginning of the window buffer.

Public Interfaces

• Changes to the Evictor interface :  addition two new methods evictBefore and evictAfter, removal of the existing evict method
• Corresponding changes to CountEvicot, DeltaEvictor
• Renaming of the existing TimeEvictor as ProcessingTimeEvictor and addition of new class EventTime evictor
• New overloaded method CountEvictor.of() that takes an additional parameter doEvictAfter, which decides whether to do eviction after the WindowFunction.
• Similar overloaded methods in DeltaEvictor, ProcessingTimeEvictor and EventTimeEvictor.

Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.

A public interface is any change to the following:

• DataStream and DataSet API, including classes related to that, such as StreamExecutionEnvironment

Proposed Changes

Describe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users? 
• If we are changing behavior how will we phase out the older behavior? 
• If we need special migration tools, describe them here.
• When will we remove the existing behavior?

Test Plan

Describe in few sentences how the FLIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.