Versions Compared

Old Version 34

changes.mady.by.user Weijie Guo

Saved on

compared with

New Version 35

changes.mady.by.user Weijie Guo

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
languagejava
titleCollector.java
/** This class take response for collecting data to output stream. */
public interface Collector<OUT> {
    /**
     * Collect record to output stream.
     *
     * @param record to be collected.
     */
    void collect(OUT record);      
    
    /**
     * Overwrite the timestamp of this record and collect it to output stream.
     *
     * @param record to be collected.
     * @param timestamp of the processed data.
     */
    void collectAndOverwriteTimestamp(OUT record, long timestamp); 
}

NonPartitionedContext

Sometimes it is not possible to decide on the current partition in the context of the processing. For example, when processing the input records from the broadcast edge. Therefore, we introduce a mechanism to process all partitions.

Code Block
languagejava
titleCollector.java
/**
 * This interface represents the context associated with all operations must be applied to all
 * partitions.
 */ public interface NonPartitionedContext<OUT> {
    /**
     * Apply a function to all partitions. For keyed stream, it will apply to all keys. For
     * non-keyed stream, it will apply to single partition.
     */
    void applyToAllPartitions(ApplyPartitionFunction<OUT> applyPartitionFunction);
}

/** A function to be applied to all partitions . */
@FunctionalInterface
public interface ApplyPartitionFunction<OUT> {
    /**
     * The actual method to be applied to each partition.
     *
     * @param collector to output data.
     * @param ctx runtime context in which this function is executed.
     */
    void apply(Collector<OUT> collector, RuntimeContext ctx) throws Exception;
}

OneInputStreamProcessFunction

Code Block
languagejava
titleOneInputStreamProcessFunction.java


/**
 * This class contains all logical related to process records from single input. interface represents the context associated with all operations must be applied to all
 * partitions with two outputs.
 */
public interface OneInputStreamProcessFunction<INTwoOutputNonPartitionedContext<OUT1, OUT> extends ProcessFunctionOUT2> {
    /**
     * Process record and emit data through {@link Collector}.
     * Apply a function to all partitions. For keyed stream, it will apply to all keys. For
     * @param recordnon-keyed stream, it will apply to processsingle partition.
     */
 @param  output to emit processed records.
     * @param ctx, runtime context in which this function is executed.
     */
    void processRecord(IN record, Collector<OUT> output, RuntimeContext ctx) throws Exception;

void applyToAllPartitions(TwoOutputNonPartitionedContext<OUT1, OUT2> applyPartitionFunction);
}

/** A function to be applied to all partitions with two outputs.. */
@FunctionalInterface
public interface TwoOutputApplyPartitionFunction<OUT1, OUT2> {
    /**
     * ThisThe isactual amethod life-cycleto methodbe indicatesapplied thatto thiseach functionpartition.
 will no longer receive any input*
     * data @param firstOutput to emit record to first output.
     * @param secondOutput to emit record to second output.
     * @param ctx, theruntime context in which this function is executed.
     */
    default void endInput(NonPartitionedContextapply(Collector<OUT1> firstOutput, Collector<OUT2> secondOutput, RuntimeContext ctx) {}
            throws Exception;
}

...

OneInputStreamProcessFunction

Code Block
languagejava
titleTwoInputStreamProcessFunctionOneInputStreamProcessFunction.java
/** This class contains all logical related to process records from twosingle input. */
public interface TwoInputStreamProcessFunction<IN1OneInputStreamProcessFunction<IN, IN2, OUT> extends ProcessFunction {
    /**
     * Process record from first input and emit data through {@link Collector}.
     *
     * @param record to process.
     * @param output to emit processed records.
     * @param ctx, runtime context in which this function is executed.
     */
    void processRecordFromFirstInputprocessRecord(IN1IN record, Collector<OUT> output, RuntimeContext ctx)
            throws Exception;

    /**
     * ProcessThis recordis froma secondlife-cycle inputmethod andindicates emitthat datathis throughfunction {@link Collector}.
     *will no longer receive any input
     * @param record to processdata.
     * @param output to emit processed records.
     * @param ctx, runtimethe context in which this function is executed.
     */
    default void processRecordFromSecondInputendInput(IN2 record, Collector<OUT> output, RuntimeContext NonPartitionedContext ctx) {}
}

TwoInputStreamProcessFunction

Code Block
languagejava
titleTwoInputStreamProcessFunction.java
/** This class contains all logical related to process records from two input. */
public interface TwoInputStreamProcessFunction<IN1, IN2, OUT> extends ProcessFunction {
    /**
            throws Exception;

    /**
     * This is a life-cycle method indicates that this function will no longer receive any data from
     * Process therecord from first input.
 and emit data  *through {@link Collector}.
     *
 @param ctx, the context in* which@param thisrecord functionto is executedprocess.
     */
 @param output to defaultemit void endFirstInput(NonPartitionedContext ctx) {}

    /**processed records.
     * This@param isctx, aruntime life-cyclecontext methodin indicateswhich that this function willis noexecuted.
 longer receive any data from*/
    void * the second input.
  processRecordFromFirstInput(IN1 record, Collector<OUT> output, RuntimeContext ctx)
   *
     * @param ctx, the context in which this function is executed.throws Exception;

    /**
     */
 Process record from defaultsecond void endSecondInput(NonPartitionedContext ctx) {}
}

BroadcastTwoInputStreamProcessFunction

Code Block
languagejava
titleBroadcastTwoInputStreamProcessFunction
/**
 * This class contains all logical related to process records from a broadcast stream and a
 * non-broadcast stream.
 */
public interface BroadcastTwoInputStreamProcessFunction<IN1, IN2, OUT> extends ProcessFunction {
    /**
     * Process record from non-broadcast input and emit data through {@link Collector}.
     *
     * @param record to process.input and emit data through {@link Collector}.
     *
     * @param record to process.
     * @param output to emit processed records.
     * @param ctx, runtime context in which this function is executed.
     */
    void processRecordFromSecondInput(IN2 record, Collector<OUT> output, RuntimeContext ctx)
            throws Exception;

    /**
     * This is a life-cycle method indicates that this function will no longer receive any data from
     * @param output to emit processed records. the first input.
     *
     * @param ctx, runtimethe context in which this function is executed.
     */
    default void processRecordFromNonBroadcastInputendFirstInput(IN1 record, Collector<OUT> output, RuntimeContext NonPartitionedContext ctx)
            throws Exception; {}

    /**
     * ProcessThis recordis froma broadcast input.In general, the broadcast side is not allowed to
     * manipulate state and output data because it corresponds to all partitions instead of a singlelife-cycle method indicates that this function will no longer receive any data from
     * the second input.
     *
     * partition.@param But you could use broadcastctx, the context toin processwhich allthis thefunction partitionsis at onceexecuted.
     */
    default *void endSecondInput(NonPartitionedContext ctx) {}
}

BroadcastTwoInputStreamProcessFunction

Code Block
languagejava
titleBroadcastTwoInputStreamProcessFunction
/**
 * This class contains all logical related to process records from a broadcast stream and a
 * non-broadcast stream.
 */
public interface BroadcastTwoInputStreamProcessFunction<IN1, IN2, OUT> extends ProcessFunction {
@param record to process.
     * @param ctx, the context in which this function is executed.
     */
    void processRecordFromBroadcastInput(IN2 record, NonPartitionedContext<OUT> ctx) throws Exception;

    /**
     * ThisProcess isrecord afrom lifenon-cyclebroadcast methodinput indicatesand thatemit thisdata functionthrough will no longer receive any data from{@link Collector}.
     *
     * @param therecord non-broadcastto inputprocess.
     * @param output to emit processed records.
     * @param ctx, theruntime context in which this function is executed.which this function is executed.
     */
    void processRecordFromNonBroadcastInput(IN1 record, Collector<OUT> output, RuntimeContext ctx)
     */
    default void endNonBroadcastInput(NonPartitionedContext ctx) {}throws Exception;

    /**
     * ThisProcess record from broadcast input.In general, the broadcast side is anot life-cycle method indicates that this function will no longer receive any data from
     * the broadcast input.allowed to
     * manipulate state and output data because it corresponds to all partitions instead of a single
     *
 partition. But you could *use @parambroadcast ctx,context theto contextprocess in whichall thisthe functionpartitions isat executedonce.
     */
    default void endBroadcastInput(NonPartitionedContext<OUT> ctx) {}
}

TwoOutputStreamProcessFunction

Code Block
languagejava
titleTwoOutputStreamProcessFunction.java
/** This@param class contains all logical related record to process and.
 emit records to two outputs. */
public interface@param TwoOutputStreamProcessFunction<INctx, OUT1,the OUT2>context extendsin ProcessFunctionwhich {
this function   /**is executed.
     */
 Process and emit recordvoid to the first/second output through {@link Collector}s.processRecordFromBroadcastInput(IN2 record, NonPartitionedContext<OUT> ctx) throws Exception;

     /**
     * @paramThis recordis to process.
     * @param output1 to emit processed records to the first output.a life-cycle method indicates that this function will no longer receive any data from
     * @paramthe output2 to emit processed records to the second output.non-broadcast input.
     *
     * @param ctx, runtimethe context in which this function is executed.
     */
    default void processRecord(
            IN record, Collector<OUT1> output1, Collector<OUT2> output2, RuntimeContextendNonBroadcastInput(NonPartitionedContext ctx); {}

    /**
     * This is a life-cycle method indicates that this function will no longer receive any data from
     * the broadcast input.
     * data.
     *
 
     * @param ctx, the context in which this function is executed.
     */
    default void endBroadcastInput(NonPartitionedContext<OUT> ctx) {}
}

TwoOutputStreamProcessFunction

Code Block
languagejava
titleTwoOutputStreamProcessFunction.java
/** This class contains all logical related to process and emit records to two outputs. */
public interface TwoOutputStreamProcessFunction<IN, OUT1, OUT2> extends ProcessFunction {
    /**    * @param ctx, the context in which this function is executed.
     */
 Process and emit record default void endInput(TwoOutputNonPartitionedContext ctx) {}
}

/**
 * This interface represents the context associated with all operations must be applied to all
 * partitions with two outputs.
 */
public interface TwoOutputNonPartitionedContext<OUT1, OUT2> {
    /**to the first/second output through {@link Collector}s.
     *
     * @param record to process.
     * @param output1 to emit processed records to the first output.
     * Apply@param a functionoutput2 to allemit partitions.processed Forrecords keyedto stream,the it will apply to all keys. Forsecond output.
     * non-keyed@param streamctx, runtime context itin willwhich applythis tofunction singleis partitionexecuted.
     */
    void applyToAllPartitionsprocessRecord(TwoOutputNonPartitionedContext<OUT1,
  OUT2> applyPartitionFunction);
}

/** A function to be applied to all partitions withIN two outputs.. */
@FunctionalInterface
public interface TwoOutputApplyPartitionFunction<OUT1, OUT2> {
 record, Collector<OUT1> output1, Collector<OUT2> output2, RuntimeContext ctx);

   /**
     * This is Thea actuallife-cycle method toindicates bethat appliedthis tofunction eachwill partition.
no longer receive any  *input
     * @param firstOutput to emit record to first outputdata.
     * @param secondOutput to emit record to second output.
     * @param ctx, runtimethe context in which this function is executed.
     */
    default void applyendInput(Collector<OUT1> firstOutput, Collector<OUT2> secondOutput, RuntimeContext TwoOutputNonPartitionedContext ctx)
            throws Exception; {}
}

RuntimeContext contains information about the context in which process functions are executed. It is currently just an empty interface but will be expanded later, such as supporting access to state, registering timers, etc. This part will be elaborated in the subsequent sub-FLIPs.

...