Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • `jobmanager.heap.size` option is deluding for the containerised environments (Yarn/Mesos/Kubernetes) because it does not represent the JM’s JVM heap size but the total process memory consumed by the Flink JVM process including container cut-off. It is used to set the JM container memory size requested in a containerised environment.
  • The purpose of the container cut-off can be also confusing on its own, the main use cases for it are:
    • Direct memory usage by Flink or user code dependencies (there are certain cases where user code is run during the job start up)
    • JVM Metaspace
    • Other JVM overhead

...

The JVM heap size is set as corresponding JVM arguments (-Xms and  -Xmx) when the JM process is started in standalone scripts or container startup script. There can be the following possible sources of the JVM heap consumption in JM:

  • Flink framework
  • User code running during job submission in certain cases or in checkpoint completion callbacks
  • Job Cache


The size of JVM heap is mostly driven by the amount of running jobs, their structure and requirements for the mentioned user code.

...