THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
| Page properties | ||
|---|---|---|
|
Table of Contents
Status
Current state: "Under Discussion"
...
...
...
|
...
...
|
Released:Table of Contents
Motivation
This FLIP suggests aligning the memory model and configuration for Job Manager (JM) with the recently introduced memory model of Task Manager (TM) in FLIP-49 .
The memory model of JM does not need to be as extensive as the TM one. A lot of motivation points in FLIP-49 are not applicable here. Nonetheless, apart of aligning two memory models, there are couple of explicit issues with the current memory settings of JM:
...
The size of JVM Metaspace is mostly driven by the requirements for the mentioned user code. Therefore, it may be hard to come up with its default value which covers all the use cases so it can be a point of tuning. Preliminary, it can be set to 128Mb256Mb.
JVM Overhead
All JVM and other off-heap memory consumption, which is not accounted for in other memory components, can be part of this JVM Overhead . This is mostly relevant for the containerised environments to derive the Total Process Memory and request the size of the JM container memory. The way of the JVM Overhead configuration can be the same as for TM. The size of the JVM Overhead can be a configured fraction of the Total Process Memory which is capped by its configured min and max values. See also ' Configuration process' further .
...
Memory component | options | Default value |
Total Process Memory | jobmanager.memory.process.size | None (“1472m” in default flink-conf.yaml ) |
Total Flink Memory | jobmanager.memory.flink.size | None |
JVM Heap | jobmanager.memory.heap.size | None |
Off-heap memory | jobmanager.memory.off-heap.size | “128m” |
JVM Metaspace | jobmanager.memory.jvm-metaspace.size | “256m” |
JVM Overhead | jobmanager.memory.jvm-overhead.min | “192m” |
jobmanager.memory.jvm-overhead.max | “1g” | |
jobmanager.memory.jvm-overhead.fraction “192m” “1g” | 0.1 |
Implementation Steps
- Introduced new options
- Introduce data structures and utilities
- Data structure to store memory sizes of JM
- Utility for calculating memory sizes from configuration
- Extend the calculation utility and BashJavaUtils with generating JVM arguments to start JM process
- Call BashJavaUtils in the standalone startup scripts and use returned JVM arguments to start JM JVM process (ClusterEntryPoint) instead of current bash code
- Use new memory calculation utility to get the Total Process Memory size and the JVM arguments to start the JM container (ClusterEntryPoint) in the containerized environment
...