Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
languagejava
titlePersistentInFlightDataStorer
linenumberstrue
collapsetruePersist
interface PersistentInFlightDataStorer extends Closeable {
	/**
	 * Appends a given buffer to the persisting storage. 
	 * <p>This method may be blocking until the data is completely persisted.
	 * <p>If this method is non-blocking, implementers must make a defensive copy of the buffer.
	 */ 
	void append(Buffer buffer) throws IOException;
 
	/**
	 * Finalizes the storage for this particular input or output.
	 * <p>For non-blocking append, this method must ensure that all data has been 
	 * successfully persisted and indicate any error.
	 */
	@Override
	void close() throws IOException;
}

...

Code Block
languagejava
titlePersistentInFlightDataRetriever
linenumberstrue
collapsetruePersist
interface PersistentInFlightDataRetriever extends AvailabilityProvider {
	/**
	 * Returns the next buffer if available. This method should be non-blocking.
	 */
	Optional<Buffer> pollNext();
}

...

Code Block
languagejava
titlePersistentInFlightDataStorage
linenumberstrue
collapsetruePersist
interface PersistentInFlightDataStorage {
	PersistentInFlightDataStorer createStorer(int gateId);
	PersistentInFlightDataRetriever createRetriever(int gateId);
}

...

Furthermore, the state containing the persisted data of an input/output should also contain a small header. The header consists of a version number identifying the used checkpoint format. For the first version (described above), the number is ‘1’ and gives us the option for backward compatible changes. See Figure 4 2 for an overview over all involved headers. Note that the current buffer format including record header remains untouched and is just shown for completeness.

...

  • No guarantees for non-keyed data (Kafka way) was rejected because we knew some users (currently) depend on these guarantees.
  • Explicitly stating partitioning by users (reinterpret stream) would be a possible alternative for special cases where the current proposal fails.
  • No scaling for non-keyed data would make the work stealing of FLIP-27 hard to implement beyond the source.

...