THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Table of Contents |
---|
Motivation
This FLIP aims to solve several problems/shortcomings in the current streaming source interface (SourceFunction
) and simultaneously to unify the source interfaces between the batch and streaming APIs. The shortcomings or points that we want to address are:
...
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
public interface SourceOutput<E> extends WatermarkOutput { void emitRecord(E record, String splitId); void emitRecord(E record, String splitId, long timestamp); } |
SourceReaderContext
...
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
/** * Emit a record to the downstream. * * @param <E> the type of the record emitted by the {@link SplitReader} * @param <T> the type of records that are eventually emitted to the {@link SourceOutput}. * @param <SplitStateT> the mutable type of split state. */ public interface RecordEmitter<E, T, SplitStateT> { /** * Process and emit the records to the {@link RecordEmitter.OutputSourceOutput}. A few recommendations to the implementation * are following: * * <ul> * <li>The method maybe interrupted in the middle. In that case, the same set of records will be passed * to the record emitter again later. The implementation needs to make sure it reades * <li> * </ul> * * @param element The intermediate element read by the SplitReader. * @param splitId The ID of the split the element was consumed from. * @param output The output to which the final records are emit to. * @param splitState The state of the split. */ void emitRecord(E element, StringSourceOutput<T> splitId, Output<T> output, SplitStateT splitState) throws Exception; /** * The interface for RecordEmitter output. * * @param <T> the type of the record emit */ interface Output<T> { /** * Emit an element without a timestamp. Equivalent to {@link #collect(Object, Long) collect(timestamp, null)}; * * @param element */ void collect(T element) throws Exception; /** * Emit an element with timestamp. * * @param element * @param timestamp */ void collect(T element, Long timestamp) throws Exception; } } |
Public interface from RPC gateway
...