THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Sometimes it is not possible to decide on the current partition in the context of the processing. For example, when processing the input records from the broadcast edge. Therefore, we introduce a mechanism to process all partitions.
Note: RuntimeContext contains information about the context in which process functions are executed. It is currently just an empty interface but will be expanded later, such as supporting access to state, registering timers, etc. This part will be elaborated in the subsequent sub-FLIPs.
| Code Block | ||||
|---|---|---|---|---|
| ||||
/**
* This interface represents the context associated with all operations must be applied to all
* partitions.
*/
public interface NonPartitionedContext<OUT> extends RuntimeContext {
/**
* Apply a function to all partitions. For keyed stream, it will apply to all keys. For
* non-keyed stream, it will apply to single partition.
*/
void applyToAllPartitions(ApplyPartitionFunction<OUT> applyPartitionFunction);
}
/** A function to be applied to all partitions . */
@FunctionalInterface
public interface ApplyPartitionFunction<OUT> {
/**
* The actual method to be applied to each partition.
*
* @param collector to output data.
* @param ctx runtime context in which this function is executed.
*/
void apply(Collector<OUT> collector, RuntimeContext ctx) throws Exception;
}
/**
* This interface represents the context associated with all operations must be applied to all
* partitions with two outputs.
*/
public interface TwoOutputNonPartitionedContext<OUT1, OUT2> extends RuntimeContext {
/**
* Apply a function to all partitions. For keyed stream, it will apply to all keys. For
* non-keyed stream, it will apply to single partition.
*/
void applyToAllPartitions(TwoOutputNonPartitionedContext<OUT1, OUT2> applyPartitionFunction);
}
/** A function to be applied to all partitions with two outputs.. */
@FunctionalInterface
public interface TwoOutputApplyPartitionFunction<OUT1, OUT2> {
/**
* The actual method to be applied to each partition.
*
* @param firstOutput to emit record to first output.
* @param secondOutput to emit record to second output.
* @param ctx runtime context in which this function is executed.
*/
void apply(Collector<OUT1> firstOutput, Collector<OUT2> secondOutput, RuntimeContext ctx)
throws Exception;
} |
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
/** This class contains all logical related to process and emit records to two outputs. */
public interface TwoOutputStreamProcessFunction<IN, OUT1, OUT2> extends ProcessFunction {
/**
* Process and emit record to the first/second output through {@link Collector}s.
*
* @param record to process.
* @param output1 to emit processed records to the first output.
* @param output2 to emit processed records to the second output.
* @param ctx, runtime context in which this function is executed.
*/
void processRecord(
IN record, Collector<OUT1> output1, Collector<OUT2> output2, RuntimeContext ctx);
/**
* This is a life-cycle method indicates that this function will no longer receive any input
* data.
*
* @param ctx, the context in which this function is executed.
*/
default void endInput(TwoOutputNonPartitionedContext ctx) {}
} |
...
We can see that each process function provides the life-cycle hook for endInput. The runtime engine will call back this method after processing all data of this input, providing the final opportunity to send data to downstream. This is crucial for implementing something like full-aggregation window.
...