THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Status
Current state: [Under DiscussionIn Progress]
Discussion thread: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
JIRA: Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-1414419661
Released: <Flink Version>
...
This includes making JobManager's memory-related metrics available in the UI. Additionally, the configuration parameters should be exposed similar to the TaskManager's overview (see FLIP-102: Add More Metrics to TaskManager).
JVM Metrics
These JVM metrics are exposed and can be used through the metrics REST API.
| JVM | Metric | Used key | Total key |
|---|---|---|---|
| Heap |
Flink Compose0
user conf key1
configuration key2
metric max3
metric used3
Heap
| Status.JVM.Memory.Heap |
| Used | Max | |
| Direct | Status.JVM.Memory. |
| Direct | Used |
Off-Heap
jobmanager.memory.off-heap.size| Max | |
| Mapped | Status.JVM.Memory. |
| Mapped | MemoryUsed | TotalCapacity |
| NonHeap | Status.JVM.Memory.NonHeap |
JVM MetaSpace
| MemoryUsed | TotalCapacity |
| Metaspace | Status.JVM.Memory.Metaspace |
| Used | Max |
Memory Configuration
Flink's memory model (as described in org.apache.flink.runtime.jobmanager.JobManagerProcessSpec) can be mapped to the following Flink configuration parameters. There are a few that have a correlating Flink metric.
| Flink Memory Model | Flink configuration1 | REST API2 | Metric | Used key | Total key |
|---|---|---|---|---|---|
| Heap | jobmanager.memory.heap.size | ??? | Status.JVM.Memory.Heap | Used | Max |
| Off-Heap | jobmanager.memory.off-heap.size | ??? | - | - | - |
| JVM Metaspace |
jobmanager.memory.jvm-metaspace.size | ??? | Status.JVM.Memory.Metaspace | Used | Max | |||||||||
| JVM Overhead | jobmanager.memory.jvm-overhead.min |
memoryConfiguration.jvmOverhead
Status.JVM.Memory.NonHeap.Max - Status.JVM.Memory.Metaspace.Max
| ??? | - | - | - |
jobmanager.memory.jvm-overhead.max |
0 The partitioning how it is described in org.apache.flink.runtime.clusterframework.TaskExecutorProcessSpec.
1 These are the configuration parameters used in the Flink configuration.
2 These are the Json paths to address the properties in the HTTP REST API response. Additionally, memoryConfiguration.totalFlinkMemory and totalProcessMemory are exposed through the REST API.
3 The metrics which are exposed through the metrics endpoint. Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-19662
Frontend Design
Display metrics inside JVM, outside JVM and GC
...
The API can be used to retrieve the metrics for the JobManager: http://localhost:8081/jobmanager/metrics?get=Status.JVM.Memory.Heap.Max,Status.JVM.Memory.Heap.Used,Status.JVM.Memory.NonHeap.Max,Status.JVM.Memory.NonHeap.Used,Status.JVM.Memory.Metaspace.Max,Status.JVM.Memory.Metaspace.Used
Memory Configuration
Test Plan
Covered by unit tests.