Page History

...

That means, if a checkpoint succeeds but we do not receive the success event, the infight instant will span two/(or more) checkpoints.

The Semantics

In order to improve the throughput, The function process thread does not block data buffering after the checkpoint thread starts flushing the existing data buffer. So there is possibility that the next checkpoint batch was written to current checkpoint. When a checkpoint failure triggers the write rollback, there may be some duplicate records (e.g. the eager write batch), the semantics is still correct using the UPSERT operation.

We may improve it to be exactly-once once the write supports write record one-by-one instead of batches (The write shift the file handles on checkpointing).

Fault Tolerance

We do not block when BucketWriter finish a checkpoint buffer flush, during the time interval until to the checkpoint completes, there probably be other mini-batch buffer (name it A) flush out, when a checkpoint timeout, we would re-consume the A buffer and flush it out, thus the buffer A data is written duplicately.

The operator coordinator checks and commits the last instant then starts a new one when a checkpoint finished successfully. The operator rolls back the written data and throws to trigger a failover when any error occurs. This means one Hoodie instant may span one or more checkpoints(some checkpoints notifications may be skipped). If a checkpoint timed out, the next checkpoint would help to rewrite the left buffer data (clean the buffer in the last step of the #flushBuffer method).

The operator coordinator would try several times when committing the write status.

# STEP4: A New Index for Flink

...