Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Reverted from v. 23

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Table of Contents

Motivation

This FLIP aims to solve several problems/shortcomings in the current streaming source interface (SourceFunction) and simultaneously to unify the source interfaces between the batch and streaming APIs. The shortcomings or points that we want to address are:

...

Code Block
languagejava
titleSourceOutput
linenumberstrue
public interface SourceOutput<E> extends WatermarkOutput {

	void emitRecord(E record, String splitId);

	void emitRecord(E record, String splitId, long timestamp);
}

SourceReaderContext

...

Code Block
languagejava
titleRecordEmitter
linenumberstrue
/**
 * Emit a record to the downstream.
 *
 * @param <E> the type of the record emitted by the {@link SplitReader}
 * @param <T> the type of records that are eventually emitted to the {@link SourceOutput}.
 * @param <SplitStateT> the mutable type of split state.
 */
public interface RecordEmitter<E, T, SplitStateT> {

	/**
	 * Process and emit the records to the {@link RecordEmitter.OutputSourceOutput}. A few recommendations to the implementation
	 * are following:
	 *
	 * <ul>
	 * 	<li>The method maybe interrupted in the middle. In that case, the same set of records will be passed
	 * 	to the record emitter again later. The implementation needs to make sure it reades
	 * 	<li>
	 * </ul>
	 *
	 * @param element The intermediate element read by the SplitReader.
     * @param splitId The ID of the split the element was consumed from.
	 * @param output The output to which the final records are emit to.
	 * @param splitState The state of the split.
	 */
	void emitRecord(E element, StringSourceOutput<T> splitId, Output<T> output, SplitStateT splitState) throws Exception;

	/**
	 * The interface for RecordEmitter output.
	 *
	 * @param <T> the type of the record emit
	 */
	interface Output<T> {
		/**
		 * Emit an element without a timestamp. Equivalent to {@link #collect(Object, Long) collect(timestamp, null)};
		 *
		 * @param element
		 */
		void collect(T element) throws Exception;

		/**
		 * Emit an element with timestamp.
		 *
		 * @param element
		 * @param timestamp
		 */
		void collect(T element, Long timestamp) throws Exception;
	}
}

Public interface from RPC gateway

...