Status
Current state: [Under Discussion]
Discussion thread: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
JIRA:
Released: <Flink Version>
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Aligning the overview page of JobManager with the Flink's new memory model (FLIP-116).
Proposed Changes
This includes making JobManager's memory-related metrics available in the UI. Additionally, the configuration parameters should be exposed similar to the TaskManager's overview (see FLIP-102: Add More Metrics to TaskManager).
JVM | Flink Compose | user conf key1 | configuration key2 | metric max3 | metric used3 |
Heap | Framework Heap | memoryConfiguration.frameworkHeap | Status.JVM.Memory.Heap.Max | Status.JVM.Memory.Heap.Used | |
Off-Heap | Framework Off-Heap | jobmanager.memory.off-heap.size | memoryConfiguration.frameworkOffHeap | Status.JVM.Memory.NonHeap.Max | Status.JVM.Memory.NonHeap.Used |
JVM MetaSpace | memoryConfiguration.jvmMetaspace | Status.JVM.Memory.Metaspace.Max | Status.JVM.Memory.Metaspace.Used
| ||
JVM Overhead | memoryConfiguration.jvmOverhead | Status.JVM.Memory.NonHeap.Max - Status.JVM.Memory.Metaspace.Max | Status.JVM.Memory.NonHeap.Used - Status.JVM.Memory.Metaspace.Used | ||
1 These are the configuration parameters used in the Flink configuration.
2 These are the Json paths to address the properties in the HTTP REST API response. Additionally, memoryConfiguration.totalFlinkMemory
and totalProcessMemory
are exposed through the REST API.
3 The metrics which are exposed through the metrics endpoint.
Frontend Design
Display metrics inside JVM, outside JVM and GC
REST API proposal:
The API provided now could only get the metric key without a value.
- add JobManager metrics in MetricStore
- show JobManager memory and Garbage collectors
- after FLINK-9741 finished, show it according to flip-6
- url: /jobmanager/metrics/info
- response:
{ "type" : "object", "id" : "urn:jsonschema:org:apache:flink:runtime:rest:messages:cluster:JobManagerMetricsInfo", "properties" : { "heapUsed" : { "type" : "integer" }, "heapCommitted" : { "type" : "integer" }, "heapMax" : { "type" : "integer" }, "nonHeapUsed" : { "type" : "integer" }, "nonHeapCommitted" : { "type" : "integer" }, "nonHeapMax" : { "type" : "integer" }, "directCount" : { "type" : "integer" }, "directUsed" : { "type" : "integer" }, "directMax" : { "type" : "integer" }, "mappedCount" : { "type" : "integer" }, "mappedUsed" : { "type" : "integer" }, "mappedMax" : { "type" : "integer" }, "garbageCollectors" : { "type" : "array", "items" : { "type" : "object", "id" : "urn:jsonschema:org:apache:flink:runtime:rest:messages:cluster:JobManagerMetricsInfo:GarbageCollectorInfo", "properties" : { "name" : { "type" : "string" }, "count" : { "type" : "integer" }, "time" : { "type" : "integer" } } } } } } |
Test Plan
Covered by unit tests.