Status

Current state: Under Discussion

Discussion thread:

JIRA:

Released:

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Streaming jobs which run for several days or longer usually experience changes in their workload during their lifetime. These changes can originate from seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs. non-holidays, sudden events or simply growing popularity of your product. Some of these changes are more predictable than others but what all have in common is that they change the resource demand of your job if you want to keep the same service quality for your customers.

Even if you can estimate an upper bound for the maximum resource requirements, it is almost always prohibitively expensive to run your job from the very beginning with max resources. Consequently, it would be great if Flink could make use of resources (TaskManagers) which become available after a job has been started. Similarly, if the underlying resource management system decides that some of the currently assigned resources are needed elsewhere, Flink should not fail but scale the job down once the resources are revoked. Such a behaviour would make Flink a nicer citizen of the respective resource manager.

Ideally, Flink would control the resource (de-)allocation. However, not every Flink deployment knows about the underlying resource management system. Moreover, it can be easier to decide on the actual resource need from the application developer's perspective (external party). Hence, we are proposing the reactive execution mode which enables Flink to react to newly available or removed resources by scaling the job up or down with the goal to use the available resources as good as possible.

The Reactive Mode allows Flink users to implement a powerful autoscaling mechanism, by having an external service monitor certain metrics, such as consumer lag, aggregate CPU utilization, throughput or latency. As soon as these metrics are above or beyond a certain threshold, additional TaskManagers can be added or removed from the Flink cluster. This could be implemented through changing the replica factor of a Kubernetes deployment, or an autoscaling group.

Proposed Changes

Overview

The proposed change builds upon the declarative resource management (FLIP-138) and the declarative scheduler (FLIP-160). With these changes, it is possible to declare a set of required resources, to start executing a job even if we haven't received all of the declared resources and to rescale the job if new resources become available. These are the ingredients we need in order to implement the reactive mode.

The reactive mode cannot know how many resources will be made available to the Flink cluster. Consequently, it needs to acquire all available resources. We can achieve this by declaring an infinite amount of required resources. This will make sure that whenever a new TaskManager registers at the ResourceManager, then the ResourceManager will assign the new slots to the JobMaster which is responsible for executing the job. Using the declarative scheduler, we will start executing the job once the set of resources has stabilised and adjust the parallelism whenever the set of resources changes. Thereby Flink will become able to make use of all resources which are available in the cluster. The definition of "stable resources" will be discussed in FLIP-160.

The way users will use this feature is by deploying a per-job or application cluster with the configuration option execution-mode  set to reactive. This option will only be allowed if the user deploys a streaming job and uses either the per-job or application cluster (dedicated cluster per job). Session clusters will not support the reactive mode.

bin/flink run -Dexecution-mode=reactive --target yarn-per-job path/to/MyJob.jar

What happens underneath when using the reactive execution mode is that the parallelism of every operator will be set to Short.MAX_VALUE. Moreover the cluster will be started using the declarative scheduler.

Limitations

Initially we only intend to support this mode for a limited subset of jobs:

• Streaming jobs only
• No support for fine grained resources
• No fixed parallelism for any of the operators

Moreover, we only intend to support this mode for per-job and application clusters for the time being. This will exclude the problem of how to distribute cluster resources among multiple jobs whose requirements have not been satisfied.

The parallelism of individual operators in a streaming job will be determined by the DeclarativeScheduler. It is not configurable by the user (except that the per-operator maxParallelism is the upper bound for the parallelism determined). The maxParallelism is bounded by Short.MAX_VALUE (32767).

Implementation

On the JobMaster side, the Reactive Mode is build on top of the declarative scheduler (FLIP-160). In the first implementation of the scheduler, the implementations for the SlotAllocator and ScalingPolicy will not be configurable, and hardwired to Reactive Mode specific implementations. These implementations will expose a minimal set of configuration options, which are described in FLIP-160.

Activation by the user

The reactive mode can be activated through the new "execution-mode" configuration option (that can be passed as a -D command line argument, or through flink-conf.yaml).

If a streaming job is submitted, and execution-mode=reactive, the new declarative scheduler will be configured. The scheduler will internally change the JobGraph so that the parallelism / maxParallelism of Tasks is set to Short.MAX_VALUE.

Depending on initial user feedback, we might introduce a pre-flight sanity check into the StreamGraphGenerator, that checks if the streaming job is suitable for reactive mode. Similarly, a warning should be printed if a session cluster receives a reactive mode job.

Example with custom config:

bin/flink run -Dexecution-mode=reactive -Dpipeline.declarative-scheduler.reactive.min-execution-time="10 min" --target yarn-per-job path/to/MyJob.jar

We will also leave the option to manually configure reactive mode for advanced users. They can manually set a maximum parallelism for their streaming job, and configure a different scheduler.

Configuration

tbd – pending review of ScalingPolicy.

Compatibility, Deprecation, and Migration Plan

The reactive mode is a new feature which the user needs to explicitly activate. Hence, there is no migration needed.

Test Plan

The new execution mode should be covered by end-to-end tests which ensure the correct functioning of the reactive mode. Moreover, we should try to find open source user who are willing to try this feature out in order to gather feedback on its ergonomics and overall usefulness.

Future Extensions

Non-homogeneous scaling

In the first implementation, scaling of individual operators is not controllable by the user. It is determined by the DeclarativeScheduler, more specifically, the SlotAllocator component. This can lead to situations where the Scheduler determines a parallelism for certain operators, that is not optimal (for example a Kafka consumer having more instances than available partitions; or an operator that will anyways only run with parallelism = 1).

An extension to ReactiveMode would be giving the user more control over the scaling behavior of individual operators. Ideas for this:

• Do the scaling proportional to the initially configured parallelism, capped by the maxParallelism.
• Allow the user to define three parallelism: min / target / max. If min parallelism can not be satisfied, fail.
• Allow the user to implement a callback, that determines the parallelism for the scheduler.

Communication with external systems and customizable scaling behavior

The "ScalingPolicy" described in FLIP-160 is currently not exposed to the user, thus, scaling decisions are only customizable based on the provided configuration options.

By allowing users to implement a custom ScalingPolicy, much more customizations are possible:

• you can make calls to external system (such as a central "Flink Clusters Management Service") for scaling decisions.
• users can implement extended scaling policies, exposing more configuration parameters than just "min-executing-time" and "additional-parallelism".
• For large-state streaming jobs, it might make sense to only allow to scale up within a certain timeframe after a completed checkpoint.

Rejected Alternatives

Activation by the user

We've considered two other alternatives for the activation of reactive mode:

• Alternative 1: Introduction of a configuration parameter for the client side (setting the parallelism, checking of conditions) and on the server side (scheduler type). This alternative was rejected due to its complexity for the user.
• Alternative 2: Users have to manually set a high parallelism, ensure that the job is suitable for the declarative scheduler and configure the declarative scheduler. This alternative was rejected due to its poor user experience.