THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Status
...
| Page properties | |
|---|---|
|
...
...
...
|
...
|
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
Aligning the overview page of JobManager with the Flink's new memory model (FLIP-116). Additionally, we want to align the JobManager's details page containing memory-related information with the solution proposed in FLIP-102.
Proposed Changes
This includes making JobManager's memory-related metrics available in the UI. Additionally, the effective configuration parameters should be exposed similar to the TaskManager's overview (see FLIP-102: Add More Metrics to TaskManager).
JVM Metrics
These JVM metrics are exposed and can be used through the JobManager's metrics REST API.
Flink Compose0
user conf key1
configuration key2
metric max3
metric used3
Framework Heap
memoryConfiguration.frameworkHeap
| JVM | Metric | Used key | Total key |
|---|---|---|---|
| Heap |
| Status.JVM.Memory.Heap |
| Used | Max | |
| Direct | Status.JVM.Memory. |
| Direct | Used |
Off-Heap
jobmanager.memory.off-heap.size| Max | |
| Mapped | Status.JVM.Memory. |
| Mapped | MemoryUsed | TotalCapacity |
| NonHeap | Status.JVM.Memory.NonHeap |
JVM MetaSpace
| MemoryUsed | TotalCapacity |
| Metaspace | Status.JVM.Memory.Metaspace |
| Used | Max |
Memory Configuration
Flink's memory model (as described in org.apache.flink.runtime.jobmanager.JobManagerProcessSpec) can be mapped to the following Flink configuration parameters. There are a few that have a correlating Flink metric.
| Flink Memory Model | Flink configuration1 | Effective Configuration REST API2 | Metric3 | Used key | Total key |
|---|---|---|---|---|---|
| Heap | jobmanager.memory.heap.size | jobmanager.memory.heap.size | Status.JVM.Memory.Heap | Used | Max |
| Off-Heap | jobmanager.memory.off-heap.size | jobmanager.memory.off-heap.size | - | - | - |
| JVM Metaspace |
jobmanager.memory.jvm-metaspace.size | jobmanager.memory.jvm-metaspace.size | Status.JVM.Memory.Metaspace | Used | Max | |||||||||
| JVM Overhead | jobmanager.memory.jvm-overhead.min |
| jobmanager. |
| memory.jvm-overhead.min/ jobmanager.memory.jvm-overhead.max4 | - | - | - |
Status.JVM.Memory.NonHeap.Max - Status.JVM.Memory.Metaspace.Max
jobmanager.memory.jvm-overhead.max |
...
1 These are the configuration parameters used in the Flink configuration.
2 These are the Json paths to address the properties in the HTTP config parameters exposed through the cluster config REST API response. Additionally, memoryConfiguration.totalFlinkMemory and totalProcessMemory are exposed through the REST API.. Their names matching the actual Flink config as the effective configuration is generated out of the passed configuration. Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-19662
3 The metrics which are exposed through the metrics endpointthe JobManager's metrics REST API.
4 min and max are having the same value.
Frontend Design
Display metrics inside JVM, outside JVM and GC
Add new metrics page
REST API proposal:
Metrics
The API provided now could only get the metric key without a value.
- add JobManager metrics in MetricStore
- show JobManager memory and Garbage collectors
- after FLINK-9741 finished, show it according to flip-6
- url: /jobmanager/metrics/info
- response:
{
"type" : "object",
"id" : "urn:jsonschema:org:apache:flink:runtime:rest:messages:cluster:JobManagerMetricsInfo",
"properties" : {
"heapUsed" : {
"type" : "integer"
},
"heapCommitted" : {
"type" : "integer"
},
"heapMax" : {
"type" : "integer"
},
"nonHeapUsed" : {
"type" : "integer"
},
"nonHeapCommitted" : {
"type" : "integer"
},
"nonHeapMax" : {
"type" : "integer"
},
"directCount" : {
"type" : "integer"
},
"directUsed" : {
"type" : "integer"
},
"directMax" : {
"type" : "integer"
},
"mappedCount" : {
"type" : "integer"
},
"mappedUsed" : {
"type" : "integer"
},
"mappedMax" : {
"type" : "integer"
},
"garbageCollectors" : {
"type" : "array",
"items" : {
"type" : "object",
"id" : "urn:jsonschema:org:apache:flink:runtime:rest:messages:cluster:JobManagerMetricsInfo:GarbageCollectorInfo",
"properties" : {
"name" : {
"type" : "string"
},
"count" : {
"type" : "integer"
},
"time" : {
"type" : "integer"
}
}
}
}
}
can be used to retrieve the metrics for the JobManager: http://localhost:8081/jobmanager/metrics?get=Status.JVM.Memory.Heap.Max,Status.JVM.Memory.Heap.Used,Status.JVM.Memory.NonHeap.Max,Status.JVM.Memory.NonHeap.Used,Status.JVM.Memory.Metaspace.Max,Status.JVM.Memory.Metaspace.Used
The Metaspace metrics need to be implemented. This is going to be handled by
| Jira | ||||||
|---|---|---|---|---|---|---|
|
Memory Configuration
| Jira | ||||||
|---|---|---|---|---|---|---|
|
We want to expose the effective configuration through a new REST endpoint. We have to consider that the memory configuration depends on the type of cluster (legacy standalone vs containerized memory configuration).
...
Test Plan
Covered by unit tests.