You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Next »

Status

Current state: Under Discussion

Discussion thread: here (<- link to https://mail-archives.apache.org/mod_mbox/flink-dev/)

JIRA:[FLINK-29110] Support to mount a dynamically-created pvc for JM and TM in standalone mode with StatefulSet. - ASF JIRA (apache.org)

Released: <Flink Version>

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

The JobManager and TaskManagers of the Flink cluster currently deployed in the standalone mode are both deployed through the Deployment of Kubernetes with flink-kubernetes-operator.

However, Deployment cannot maintain the state of the component of JobManager and TaskManagers. And the Deployment does not support this at all in scenarios where PVC needs to be mounted separately for each TaskManager.

So，

Public Interfaces

The public interface is the FlinkDeployment custom resource descriptor (CRD), see below.

Proposed Changes

FlinkDeployment CRD


CR example with volumeClaimTemplate
kind: FlinkDeployment
metadata:
  namespace: default
  name: basic-example
spec:
  image: flink:1.14.3
  flinkVersion: v1_14
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "2"
  serviceAccount: flink
  jobManager:
    replicas: 1
    resource:
      memory: "2048m"
      cpu: 1
    volumeClaimTemplates: // (only needed for standalone clusters)
      - metadata:
          name: log
        spec:
          accessModes: [ "ReadWriteOnce" ]
          storageClassName: "lvm"
          resources:
            requests:
              storage: 10Gi
    podTemplate:
      apiVersion: v1
      kind: Pod
      metadata:
        name: job-manager-pod-template
      spec:
        containers:
          - name: flink-main-container
            volumeMounts:
              - name: log
                mountPath: /opt/flink/log
  taskManager:
    replicas: 4 // (only needed for standalone clusters)*     
    resource:
      memory: "2048m"
      cpu: 1
    volumeClaimTemplates: // (only needed for standalone clusters)
      - metadata:
          name: log
        spec:
          accessModes: [ "ReadWriteOnce" ]
          storageClassName: "lvm"
          resources:
            requests:
              storage: 10Gi
    podTemplate:
      apiVersion: v1
      kind: Pod
      metadata:
        name: task-manager-pod-template
      spec:
        containers:
          - name: flink-main-container
            volumeMounts:
              - name: log
                mountPath: /opt/flink/log
  mode: standalone


Compatibility, Deprecation, and Migration Plan

  • What impact (if any) will there be on existing users? 
  • If we are changing behavior how will we phase out the older behavior? 
  • If we need special migration tools, describe them here.
  • When will we remove the existing behavior?

Test Plan

Describe in few sentences how the FLIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.

  • No labels