Status
Current state: Under Discussion
Discussion thread: here (<- link to https://mail-archives.apache.org/mod_mbox/flink-dev/)
Released: <Flink Version>
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
The JobManager and TaskManagers of the Flink cluster currently deployed in the standalone mode are both deployed through the Deployment of Kubernetes with flink-kubernetes-operator.
However, Deployment cannot maintain the state of the component of JobManager and TaskManagers. And the Deployment does not support this at all in scenarios where PVC needs to be mounted separately for each TaskManager.
Using StatefulSet instead of Deployment to deploy JobManager and TaskManagers can automatically mount PVC for each pod of JobManager and TaskManagers, and can maintain the relationship between PVC and each pod.
Public Interfaces
The public interface is the FlinkDeployment custom resource descriptor (CRD), see below.
Proposed Changes
FlinkDeployment CRD
kind: FlinkDeployment metadata: namespace: default name: basic-example spec: image: flink:1.14.3 flinkVersion: v1_14 flinkConfiguration: taskmanager.numberOfTaskSlots: "2" serviceAccount: flink jobManager: replicas: 1 resource: memory: "2048m" cpu: 1 volumeClaimTemplates: // (only needed for standalone clusters) - metadata: name: log spec: accessModes: [ "ReadWriteOnce" ] storageClassName: "lvm" resources: requests: storage: 10Gi podTemplate: apiVersion: v1 kind: Pod metadata: name: job-manager-pod-template spec: containers: - name: flink-main-container volumeMounts: - name: log mountPath: /opt/flink/log taskManager: replicas: 4 // (only needed for standalone clusters)* resource: memory: "2048m" cpu: 1 volumeClaimTemplates: // (only needed for standalone clusters) - metadata: name: log spec: accessModes: [ "ReadWriteOnce" ] storageClassName: "lvm" resources: requests: storage: 10Gi podTemplate: apiVersion: v1 kind: Pod metadata: name: task-manager-pod-template spec: containers: - name: flink-main-container volumeMounts: - name: log mountPath: /opt/flink/log mode: standalone
We propose add volumeClaimTemplate to the JobManagerSpec and TaskManagerSpec to support Dynamic-created PVC mount.
JobManagerSpec class modified:
public class JobManagerSpec { /** Resource specification for the JobManager pods. */ private Resource resource; /** Number of JobManager replicas. Must be 1 for non-HA deployments. */ private int replicas = 1; /** * Volume Claim Templates for JobManager StatefulSet, it will be used to mount custom PVCs just * for standalone mode. */ private List<PersistentVolumeClaim> volumeClaimTemplates = new ArrayList<>(); /** JobManager pod template. It will be merged with FlinkDeploymentSpec.podTemplate. */ private Pod podTemplate; }
TaskManagerSpec class modified:
public class TaskManagerSpec { /** Resource specification for the TaskManager pods. */ private Resource resource; /** Number of TaskManager replicas. If defined, takes precedence over parallelism */ @SpecReplicas private Integer replicas; /** * Volume Claim Templates for TaskManager StatefulSet, it will be used to mount custom PVCs just * for standalone mode. */ private List<PersistentVolumeClaim> volumeClaimTemplates = new ArrayList<>(); /** TaskManager pod template. It will be merged with FlinkDeploymentSpec.podTemplate. */ private Pod podTemplate; }
Compatibility, Deprecation, and Migration Plan
- What impact (if any) will there be on existing users?
- If we are changing behavior how will we phase out the older behavior?
- If we need special migration tools, describe them here.
- When will we remove the existing behavior?
Test Plan
Describe in few sentences how the FLIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?
Rejected Alternatives
If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.