THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
| Page properties |
|---|
Current state: Accepted
|
...
...
|
...
|
...
|
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
The SplitEnumerator is similar to the old batch source interface's functionality of creating splits and assigning splits. It runs only once, not in parallel (but could be thought of to parallelize in the future, if necessary).
It might run on the JobManager or in a single task on a TaskManager (see below "Where to run the Enumerator").
Example:
- In the File Source , the
SplitEnumeratorlists all files (possibly sub-dividing them into blocks/ranges). - For the Kafka Source, the
SplitEnumeratorfinds all Kafka Partitions that the source should read from.
- In the File Source , the
...
- Splits are both the type of work assignment and the type of state held by the source. Assigning a split or restoring a split from a checkpoint is the same to the reader.
- Advancing the reader is a non-blocking call that returns a future.
- We build higher-level primitives on top of the main interface (see below "High-level Readers")
- We hide event-time / watermarks in the
SourceOutputand pass different source contexts for batch (no watermarks) and streaming (with watermarks).
TheSourceOutputalso abstract the per-partition watermark tracking.
The 95653748 SourceReader will run as a PushingAsyncDataInput which works well with the new mailbox threading model in the tasks, similar to the network inputs.
...
The RecordsWithSplitIds returned by the SplitReader will be passed to an 95653748 RecordEmitter one by one. The RecordEmitter is responsible for the following:
...
The interfaces used by the base implementation is covered in the section of interface for base implementation.
Failover
The state of the SplitEnumerator includes the following:
...
There was a long discussion about where to run the enumerator which we documented in the appendix. The final approach we took was very similar to option 3 with a few differences. The approach is following.
...
The event time alignment becomes easier to implement with the generic communication mechanism introduced between SplitEnumerator and SourceReader. In this FLIP we do not include this in the base implementation to reduce the complexity.
...
- The top level public interfaces.
- The interfaces introduced as a part of the base implementation of the top level public interfaces.
- The base implementation provides common functionalities required for most Source implementations. See base implementation for details.
- The RPC gateway interface change for the generic message passing mechanism.
It is worth noting that while we will try best to maintain stable interfaces, the interfaces introduced as part of the base implementation (e.g. SplitReader) is more likely to change than the top level public interface such as SplitEnumerator / SourceReader. This is primarily because we expect to add more functionality into the base implementation over time.
...