...

Current state: "Under Discussion"

Discussion thread: http://mail-archives.apache.org/mod_mbox/flink-dev/202003.mbox/%3CCAJNyZN7AJAU_RUVhnWa7r%2B%2BtXpmUqWFH%2BG0hfoLVBzgRMmAO2w%40mail.gmail.com%3E

JIRA: https://jira.apache.org/jira/browse/FLINK-16614

Released: 

Motivation

This FLIP suggests aligning the memory model and configuration for Job Manager (JM) with the recently introduced memory model of Task Manager (TM) in FLIP-49.

...

• `jobmanager.heap.size` option is deluding for the containerised environments (Yarn/Mesos/Kubernetes) because it does not represent the JM’s JVM heap size but the total process memory consumed by the Flink JVM process including container cut-off. It is used to set the JM container memory size requested in a containerised environment.
• The purpose of the container cut-off can be also confusing on its own, the main use cases for it are:
• Direct Off-heap memory usage by Flink or user code dependencies (there are certain cases where user code is run during the job start up)
• JVM Metaspace
• Other JVM overhead

• There is no way to reasonably limit direct memory JVM Direct Memory allocation, so it is not controlled by JVM. Therefore it can be hard to debug direct off-heap memory leaks and container killing because of OOM.
• Same for the JVM Metaspace size to expose possible class loading leaks.

...

The Job Cache is part of the JVM heap. It already can be configured currently by `jobstore.cache-size`. Its size should be asserted to not exceed the size of the JVM heap.

...

Off-heap Memory

The Off-heap Memory component accounts for any types of native memory usage including JVM Direct Memory. Therefore, it is also set as the corresponding JVM argument: -XX:MaxDirectMemorySize. There can be the following possible sources of the Direct Off-heap Memory consumption in JM:

• JM network communication (Akka)
• Possibly JM external Flink framework dependencies (presumably, either none or negligible at the moment)
• User code running during job submission in certain cases or in checkpoint completion callbacks

Note: Although, we say that the limit of the JVM Direct Memory is set to the size of the Off-heap Memory, the user docs should explain that this also covers non-direct memory allocation in e.g. native code. Here, we trade-off simplicity for potentially not exact JVM Direct Memory limit to avoid introducing more components.

Setting the limit for the Direct JVM Direct Memory should improve container deployment experience and debugging the JVM Direct Memory leaks.

The size of Direct Off-heap Memory is mostly driven by the network communication (amount of running jobs and their structure) and requirements for the mentioned user code. Therefore, it may be hard to come up with its default value which covers all the use cases so it can be a point of tuning. Preliminary, it can be set to 128Mb.

...

Total Flink Memory = JVM Heap + Direct Off-heap memory

JVM Overhead = Math.max(min, Math.min(max, fraction * Total Process Memory))

...

...