THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
In order to obtain the state of shuffle master, we will add the following methods to shuffle master. Before the job starts running, we will check whether the shuffle master supports taking snapshots(through method supportsBatchSnapshot). If it is not supported, we will disable Job Recovery for jobs.
Note that it's mainly designed for external/remote shuffle service, the Flink default Netty/TM shuffle is stateless and only needs to be an empty implementation.
| Code Block | ||
|---|---|---|
| ||
interface ShuffleMaster<T extends ShuffleDescriptor> extends AutoCloseable {
// other methods
/**
* Whether the shuffle master supports taking snapshot in batch scenarios, which will be used
* when enable Job Recovery. If it returns true, we will call {@link #snapshotState} to take
* snapshot, and call {@link #restoreState} to restore the state of shuffle master.
*/
default boolean supportsBatchSnapshot() {
return false;
}
default void snapshotState(CompletableFuture<byte[]> snapshotFuture) {
snapshotFuture.complete(null);
}
default void restoreState(byte[] snapshotData) {}
} |
...