...
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
interface PersistentInFlightDataStorer extends Closeable { /** * Appends a given buffer to the persisting storage. * <p>This method may be blocking until the data is completely persisted. * <p>If this method is non-blocking, implementers must make a defensive copy of the buffer. */ void append(Buffer buffer) throws IOException; /** * Finalizes the storage for this particular input or output. * <p>For non-blocking append, this method must ensure that all data has been * successfully persisted and indicate any error. */ @Override void close() throws IOException; } |
...
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
interface PersistentInFlightDataRetriever extends AvailabilityProvider { /** * Returns the next buffer if available. This method should be non-blocking. */ Optional<Buffer> pollNext(); } |
...
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
interface PersistentInFlightDataStorage { PersistentInFlightDataStorer createStorer(int gateId); PersistentInFlightDataRetriever createRetriever(int gateId); } |
...
Furthermore, the state containing the persisted data of an input/output should also contain a small header. The header consists of a version number identifying the used checkpoint format. For the first version (described above), the number is ‘1’ and gives us the option for backward compatible changes. See Figure 4 2 for an overview over all involved headers. Note that the current buffer format including record header remains untouched and is just shown for completeness.
...
- No guarantees for non-keyed data (Kafka way) was rejected because we knew some users (currently) depend on these guarantees.
- Explicitly stating partitioning by users (reinterpret stream) would be a possible alternative for special cases where the current proposal fails.
- No scaling for non-keyed data would make the work stealing of FLIP-27 hard to implement beyond the source.
...