Page History

...

1. Create the sink table in the catalog using the new API(createStageTable) according to the schema of the query result, the table is visible but not persistentrecorded in memory.
2. Submit job to the cluster.
3. After JM starts, create a table through Catalog.
4. If the job executes successfully, then make data visible.
5. If the job execution fails, then drop the sink table or delete data.(This capability requires runtime module support, such as hook, and SQL passes relevant parameters to the runtime module.)

Internal JM Hook Definition

/**
 * Hooks provided by users on job status changing.
 */
public interface JobStatusHook {

    /** When Job become CREATED status. It would only be called one time. */
    default void onCreated(JobID jobId) {}

    /** When job finished successfully. */
    default void onFinished(JobID jobId) {}

    /** When job failed finally. */
    default void onFailed(JobID jobId, Throwable throwable) {}

    /** When job get canceled by users. */
    default void onCanceled(JobID jobId) {}
}

Register JobStatusHook with StreamGraph

public class StreamGraph implements Pipeline {

    ... ...

    /** Registers the JobStatusHook. */
    void setJobStatusHook(JobStatusHook hook) {
        ...
    }
}

Supported Job Mode

Support both streaming and batch mode.

...