THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Status
...
| Page properties | |
|---|---|
|
...
...
...
|
...
|
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
Aligning the overview page of JobManager with the Flink's new memory model (FLIP-116). Additionally, we want to align the JobManager's details page containing memory-related information with the solution proposed in FLIP-102.
Proposed Changes
This includes making JobManager's memory-related metrics available in the UI. Additionally, the effective configuration parameters should be exposed similar to the TaskManager's overview (see FLIP-102: Add More Metrics to TaskManager).
JVM Metrics
These JVM metrics are exposed and can be used through the JobManager's metrics REST API.
VM
Flink Compose
user conf key
configuration key
metric max
metric used
JVM
JVM Heap
FrameWork Heap
jobmanager.memory.heap.size
memoryConfiguration.jvmHeap
| JVM | Metric | Used key | Total key |
|---|---|---|---|
| Heap |
| Status.JVM.Memory.Heap |
| Used | Max | |
| Direct | Status.JVM.Memory. |
| Direct | Used |
JVM None-Heap
JVM MetaSpace
jobmanager.memory.jvm-metaspace.size
| Max | |||
| Mapped | Status.JVM.Memory.Mapped | MemoryUsed | TotalCapacity |
| NonHeap | Status.JVM.Memory. |
| NonHeap | MemoryUsed | TotalCapacity |
| Metaspace | Status.JVM.Memory.Metaspace |
JVM Overhead
jobmanager.memory.jvm-overhead.min
memoryConfiguration.jvmOverhead
Status.JVM.Memory.NonHeap.Max
Status.JVM.Memory.NonHeap.Used
jobmanager.memory.jvm-overhead.max
other
-
-
Outside JVM
Mapped
-
-
-
| Used | Max |
Memory Configuration
Flink's memory model (as described in org.apache.flink.runtime.jobmanager.JobManagerProcessSpec) can be mapped to the following Flink configuration parameters. There are a few that have a correlating Flink metric.
| Flink Memory Model | Flink configuration1 | Effective Configuration REST API2 | Metric3 | Used key | Total key |
|---|---|---|---|---|---|
| Heap | jobmanager.memory.heap.size | jobmanager.memory.heap.size | Status.JVM.Memory. |
| Heap | Used | Max |
| Off-Heap |
Direct
jobmanager.memory.off-heap.size |
memoryConfiguration.jvmOffHeap
| jobmanager.memory.off-heap.size | - | - | - |
| JVM Metaspace | jobmanager.memory.jvm-metaspace.size | jobmanager.memory.jvm-metaspace.size |
Status.JVM.Memory. |
Metaspace |
|
|
Frontend Design
Display metrics inside JVM, outside JVM and GC
REST API proposal:
The API provided now could only get the metric key without a value.
- add JobManager metrics in MetricStore
- show JobManager memory and Garbage collectors
- after FLINK-9741 finished, show it according to flip-6
- url: /jobmanager/metrics/info
- response:
{
"type" : "object",
"id" : "urn:jsonschema:org:apache:flink:runtime:rest:messages:cluster:JobManagerMetricsInfo",
"properties" : {
"heapUsed" : {
"type" : "integer"
},
"heapCommitted" : {
"type" : "integer"
},
"heapMax" : {
"type" : "integer"
},
"nonHeapUsed" : {
"type" : "integer"
},
"nonHeapCommitted" : {
"type" : "integer"
},
"nonHeapMax" : {
"type" : "integer"
},
"directCount" : {
"type" : "integer"
},
"directUsed" : {
"type" : "integer"
},
"directMax" : {
"type" : "integer"
},
"mappedCount" : {
"type" : "integer"
},
"mappedUsed" : {
"type" : "integer"
},
"mappedMax" : {
"type" : "integer"
},
"garbageCollectors" : {
"type" : "array",
"items" : {
"type" : "object",
"id" : "urn:jsonschema:org:apache:flink:runtime:rest:messages:cluster:JobManagerMetricsInfo:GarbageCollectorInfo",
"properties" : {
"name" : {
"type" : "string"
},
"count" : {
"type" : "integer"
},
"time" : {
"type" : "integer"
}
}
}
}
}
| Used | Max | ||||||
| JVM Overhead | jobmanager.memory.jvm-overhead.min | jobmanager.memory.jvm-overhead.min/ jobmanager.memory.jvm-overhead.max4 | - | - | - | |||
jobmanager.memory.jvm-overhead.max |
1 These are the configuration parameters used in the Flink configuration.
2 These are the config parameters exposed through the cluster config REST API. Their names matching the actual Flink config as the effective configuration is generated out of the passed configuration. Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-19662
3 The metrics which are exposed through the JobManager's metrics REST API.
4 min and max are having the same value.
Frontend Design
Add new metrics page
REST API proposal:
Metrics
The API can be used to retrieve the metrics for the JobManager: http://localhost:8081/jobmanager/metrics?get=Status.JVM.Memory.Heap.Max,Status.JVM.Memory.Heap.Used,Status.JVM.Memory.NonHeap.Max,Status.JVM.Memory.NonHeap.Used,Status.JVM.Memory.Metaspace.Max,Status.JVM.Memory.Metaspace.Used
The Metaspace metrics need to be implemented. This is going to be handled by
| Jira | ||||||
|---|---|---|---|---|---|---|
|
Memory Configuration
| Jira | ||||||
|---|---|---|---|---|---|---|
|
We want to expose the effective configuration through a new REST endpoint. We have to consider that the memory configuration depends on the type of cluster (legacy standalone vs containerized memory configuration).
...
Test Plan
Covered by unit tests.