THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
However, Deployment cannot maintain the state of the component of JobManager and TaskManagers. And the Deployment does not support this at all in scenarios where PVC needs to be mounted separately for each TaskManager.
So，Using StatefulSet instead of Deployment to deploy JobManager and TaskManagers can automatically mount PVC for each pod of JobManager and TaskManagers, and can maintain the relationship between PVC and each pod.
Public Interfaces
The public interface is the FlinkDeployment custom resource descriptor (CRD), see below.
...
FlinkDeployment CRD
| Code Block | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
kind: FlinkDeployment
metadata:
namespace: default
name: basic-example
spec:
image: flink:1.14.3
flinkVersion: v1_14
flinkConfiguration:
taskmanager.numberOfTaskSlots: "2"
serviceAccount: flink
jobManager:
replicas: 1
resource:
memory: "2048m"
cpu: 1
volumeClaimTemplates: // (only needed for standalone clusters)
- metadata:
name: log
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "lvm"
resources:
requests:
storage: 10Gi
podTemplate:
apiVersion: v1
kind: Pod
metadata:
name: job-manager-pod-template
spec:
containers:
- name: flink-main-container
volumeMounts:
- name: log
mountPath: /opt/flink/log
taskManager:
replicas: 4 // (only needed for standalone clusters)*
resource:
memory: "2048m"
cpu: 1
volumeClaimTemplates: // (only needed for standalone clusters)
- metadata:
name: log
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "lvm"
resources:
requests:
storage: 10Gi
podTemplate:
apiVersion: v1
kind: Pod
metadata:
name: task-manager-pod-template
spec:
containers:
- name: flink-main-container
volumeMounts:
- name: log
mountPath: /opt/flink/log
mode: standalone |
We propose add volumeClaimTemplate to the JobManagerSpec and TaskManagerSpec to support Dynamic-created PVC mount.
JobManagerSpec class modified:
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
public class JobManagerSpec {
/** Resource specification for the JobManager pods. */
private Resource resource;
/** Number of JobManager replicas. Must be 1 for non-HA deployments. */
private int replicas = 1;
/**
* Volume Claim Templates for JobManager StatefulSet, it will be used to mount custom PVCs just
* for standalone mode.
*/
private List<PersistentVolumeClaim> volumeClaimTemplates = new ArrayList<>();
/** JobManager pod template. It will be merged with FlinkDeploymentSpec.podTemplate. */
private Pod podTemplate;
} |
TaskManagerSpec class modified:
| Code Block | ||||
|---|---|---|---|---|
| ||||
public class TaskManagerSpec { /** Resource specification for the TaskManager pods. */ private Resource resource; /** Number of TaskManager replicas. If defined, takes precedence over parallelism */ @SpecReplicas private Integer replicas; /** * Volume Claim Templates for TaskManager StatefulSet, it will be used to mount custom PVCs just * for standalone mode. */ private List<PersistentVolumeClaim> volumeClaimTemplates = new ArrayList<>(); /** TaskManager pod template. It will be merged with FlinkDeploymentSpec.podTemplate. */ private Pod podTemplate; } |
Compatibility, Deprecation, and Migration Plan
...