Status
Current state: [In Progress]
Discussion thread: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
JIRA:
Released: <Flink Version>
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Aligning the overview page of JobManager with the Flink's new memory model (FLIP-116).
Proposed Changes
This includes making JobManager's memory-related metrics available in the UI. Additionally, the configuration parameters should be exposed similar to the TaskManager's overview (see FLIP-102: Add More Metrics to TaskManager).
JVM Metrics
These JVM metrics are exposed and can be used through the JobManager's metrics REST API.
JVM | Metric | Used key | Total key |
---|---|---|---|
Heap | Status.JVM.Memory.Heap | Used | Max |
Direct | Status.JVM.Memory.Direct | Used | Max |
Mapped | Status.JVM.Memory.Mapped | MemoryUsed | TotalCapacity |
NonHeap | Status.JVM.Memory.NonHeap | MemoryUsed | TotalCapacity |
Metaspace | Status.JVM.Memory.Metaspace | Used | Max |
Memory Configuration
Flink's memory model (as described in org.apache.flink.runtime.jobmanager.JobManagerProcessSpec
) can be mapped to the following Flink configuration parameters. There are a few that have a correlating Flink metric.
Flink Memory Model | Flink configuration1 | Effective Configuration REST API2 | Metric | Used key | Total key |
---|---|---|---|---|---|
Heap | jobmanager.memory.heap.size | ??? | Status.JVM.Memory.Heap | Used | Max |
Off-Heap | jobmanager.memory.off-heap.size | ??? | - | - | - |
JVM Metaspace | jobmanager.memory.jvm-metaspace.size | ??? | Status.JVM.Memory.Metaspace | Used | Max |
JVM Overhead | jobmanager.memory.jvm-overhead.min | ??? | - | - | - |
jobmanager.memory.jvm-overhead.max |
1 These are the configuration parameters used in the Flink configuration.
2 These are the Json paths to address the properties in the HTTP REST API response.
Frontend Design
Display metrics inside JVM, outside JVM and GC
REST API proposal:
Metrics
The API can be used to retrieve the metrics for the JobManager: http://localhost:8081/jobmanager/metrics?get=Status.JVM.Memory.Heap.Max,Status.JVM.Memory.Heap.Used,Status.JVM.Memory.NonHeap.Max,Status.JVM.Memory.NonHeap.Used,Status.JVM.Memory.Metaspace.Max,Status.JVM.Memory.Metaspace.Used
Memory Configuration
Test Plan
Covered by unit tests.