Page properties | ||
---|---|---|
|
Table of Contents
Status
Current state: "Under Discussion"
...
...
...
|
...
...
|
Released: Table of Contents
Motivation
This FLIP suggests aligning the memory model and configuration for Job Manager (JM) with the recently introduced memory model of Task Manager (TM) in FLIP-49 .
...
The size of JVM Metaspace is mostly driven by the requirements for the mentioned user code. Therefore, it may be hard to come up with its default value which covers all the use cases so it can be a point of tuning. Preliminary, it can be set to 128Mb256Mb.
JVM Overhead
All JVM and other off-heap memory consumption, which is not accounted for in other memory components, can be part of this JVM Overhead . This is mostly relevant for the containerised environments to derive the Total Process Memory and request the size of the JM container memory. The way of the JVM Overhead configuration can be the same as for TM. The size of the JVM Overhead can be a configured fraction of the Total Process Memory which is capped by its configured min and max values. See also ' Configuration process' further .
...
Memory component | options | Default value |
Total Process Memory | jobmanager.memory.process.size | None (“1472m” in default flink-conf.yaml ) |
Total Flink Memory | jobmanager.memory.flink.size | None |
JVM Heap | jobmanager.memory.heap.size | None |
Off-heap memory | jobmanager.memory.off-heap.size | “128m” |
JVM Metaspace | jobmanager.memory.jvm-metaspace.size | “256m” |
JVM Overhead | jobmanager.memory.jvm-overhead.min | “192m” |
jobmanager.memory.jvm-overhead.max | “1g” | |
jobmanager.memory.jvm-overhead.fraction “192m” “1g” | 0.1 |
Implementation Steps
- Introduced new options
- Introduce data structures and utilities
- Data structure to store memory sizes of JM
- Utility for calculating memory sizes from configuration
- Extend the calculation utility and BashJavaUtils with generating JVM arguments to start JM process
- Call BashJavaUtils in the standalone startup scripts and use returned JVM arguments to start JM JVM process (ClusterEntryPoint) instead of current bash code
- Use new memory calculation utility to get the Total Process Memory size and the JVM arguments to start the JM container (ClusterEntryPoint) in the containerized environment
...