You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 13 Next »

Status

Current state: [In Progress]

Discussion thread: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html

JIRA:   Unable to render Jira issues macro, execution error.

Released: <Flink Version>

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Aligning the overview page of JobManager with the Flink's new memory model (FLIP-116). Additionally, we want to align the JobManager's details page containing memory-related information with the solution proposed in FLIP-102.

Proposed Changes

This includes making JobManager's memory-related metrics available in the UI. Additionally, the effective configuration parameters should be exposed similar to the TaskManager's overview (see FLIP-102: Add More Metrics to TaskManager).

JVM Metrics

These JVM metrics are exposed and can be used through the JobManager's metrics REST API.

JVMMetricUsed keyTotal key
HeapStatus.JVM.Memory.HeapUsedMax
DirectStatus.JVM.Memory.DirectUsedMax
MappedStatus.JVM.Memory.MappedMemoryUsedTotalCapacity
NonHeapStatus.JVM.Memory.NonHeapMemoryUsedTotalCapacity
Metaspace

Status.JVM.Memory.Metaspace Unable to render Jira issues macro, execution error.

UsedMax

Memory Configuration

Flink's memory model (as described in org.apache.flink.runtime.jobmanager.JobManagerProcessSpec) can be mapped to the following Flink configuration parameters. There are a few that have a correlating Flink metric.

Flink Memory ModelFlink configuration1Effective Configuration REST API2Metric3Used keyTotal key
Heapjobmanager.memory.heap.size???Status.JVM.Memory.HeapUsedMax
Off-Heapjobmanager.memory.off-heap.size???---
JVM Metaspacejobmanager.memory.jvm-metaspace.size???

Status.JVM.Memory.Metaspace  Unable to render Jira issues macro, execution error.

UsedMax
JVM Overheadjobmanager.memory.jvm-overhead.min???---
jobmanager.memory.jvm-overhead.max

1 These are the configuration parameters used in the Flink configuration.
2 These are the Json paths to address the properties in the HTTP REST API response.  Unable to render Jira issues macro, execution error.
3 The metrics which are exposed through the JobManager's metrics REST API.

Frontend Design

Display metrics inside JVM, outside JVM and GC

REST API proposal:

Metrics

The API can be used to retrieve the metrics for the JobManager: http://localhost:8081/jobmanager/metrics?get=Status.JVM.Memory.Heap.Max,Status.JVM.Memory.Heap.Used,Status.JVM.Memory.NonHeap.Max,Status.JVM.Memory.NonHeap.Used,Status.JVM.Memory.Metaspace.Max,Status.JVM.Memory.Metaspace.Used

The Metaspace metrics need to be implemented. This is going to be handled by  Unable to render Jira issues macro, execution error. .

Memory Configuration

Unable to render Jira issues macro, execution error.

We want to expose the effective configuration through a new REST endpoint. We have to consider that the memory configuration depends on the type of cluster (legacy standalone vs containerized memory configuration).

Test Plan

Covered by unit tests.

  • No labels