Page History

...

interface SourceReader {

    void start() throws IOException;

    CompletableFuture<?> available() throws IOException;


    ReaderStatus emitNext(SourceOutput<E> output) throws IOException;

    void addSplits(List<SplitT> splits) throws IOException;

    List<SplitT> snapshotState();
}

...

1. Sequential Single Split (File, database query, most bounded splits)
2. Multi-split multiplexed (Kafka, Pulsar, Pravega, ...)
3. Multi-split multi-threaded (Kinesis, ...)

Most of the readers implemented against these higher level building blocks would only need to implement an interface similar to this. The contract would also be that all methods except wakeup() would be called by the same thread, obviating the need for any concurrency handling in the connector.

...