Page History

...

In this section, we describe how batch mode and stream mode differ in a variety of aspects. We also describe the new behavior of Flink runtime which supports an operator to switch from batch mode to stream mode during execution of the same job.

For simplicityeasy of understanding, we assume the Flink use a simplified use-case with the following properties when describing the behavior changes:

• The job uses default configuration values for all configurations (e.g. scheduler-mode) except execution.runtime-mode

...

• (see below).
• The job has one source that switches from isBacklog=true to isBacklog=false before it reaches EOF.

Here are the definition of batch mode, stream mode and mixed mode used below:

...

• Every operator needs to support checkpoint.
• Before source operator emits isBacklog=false, checkpoint triggering is disabled.
• If any task fails when isBacklog=true, this task is restarted to re-process its input from the beginning.
• At the point when isBacklog switches to false, source operator emits RecordAttributes(isBacklog=false) and triggers an immediate checkpoint.
• Checkpoint is triggered periodically according to execution.checkpointing.interval.
• If any task fails when isBacklog=false, its pipelined region is restarted to re-process its input since the last successful checkpoint.

Extra notes: For jobs with multiple sources and execution.checkpointing.interval = 0, checkpoint triggering is enabled if and only if all sources have isBacklog=false.

As a result, suppose a source has isBacklog=true, and another source switches from isBacklog=true to isBacklog=false, the job's checkpoint would still be disabled

5) Keyed state backend

Batch mode:

...