...

Motivation

Currently users have to click on every operator and check how much data each sub-task is processing to see if there is data skew. This is particularly cumbersome and error-prone for jobs with big job graphs. Data skew is an important metric that should be more visible.

Public Interfaces

Exposed monitoring information. This is not expected to be a change in the programming interface, but rather a change in the Flink Dashboard UI and an addition to the list of available metrics. 

...

min(average_absolute_deviation(list_of_number_of_records_received_by_each_subtask) / mean(list_of_number_of_records_received_by_each_subtask) * 100, 100)

Instead of using standard deviation, I choose to use "average absolute deviation" which avoids the multiplication and the square-root operations and simplifies the metric calculation while serving the same purpose and yielding similar results.

...

Proposed metrics at Operator level:

• dataSkewPercentage: This will be used to show an overall or historical data skew score under the proposed Data Skew tab (see the UI Changes section).
  
  • The existing numRecordsIn metric can be used to build this new metric
    
• dataSkewPercentagePerSecond: This will be used to show a "live" score on the Job Graph (see the UI Changes section).
  • The existing numRecordsInPerSecond metric can be used to build this new metric

See the "rejected alternatives" section for other metrics that were considered.

...

The accumulation of "received number of records" over a long period of time can hide a recent data skew event. The same can also hide a recent fix to an existing data skew problem. Therefore the proposed metric will need to look at the change in the received number of records within a recent period, similar to the existing "Backpressure" or "Busy" metrics on the Flink Job Graph, and show a "live" data skew score.

Additional Tab to List All Operators and Their Data Skew Score in Descending Order of Their Data Skew Score

The proposed tab would sit next to the Exceptions tab as its purpose seems to me to be more similar to the Exceptions tab than other tabs. Highlighted in red blue in below screenshot.

Image Removed

Note that below screenshot/mock-up does not show the Data Skew tab next to the Exceptions tab, but the actual implementation will put it next to the Exceptions tab.

The look and feel of the proposed Data Skew tab will This FLIP does not talk in detail about how the UI of this new Data Skew tab should look. The look should be compatible with the rest of the UI. The list/table view of checkpoints under the Checkpoints tab could be used for inspiration.
The content of the proposed tab will roughly look as follows:

Image Added

It This new Data Skew tab will show the overall accumulated data skew score of the operators as opposed to current/live view proposed under the Additional "Data Skew" Metric on the Flink Job Graph section. This page tab will also contain a definition of what data skew is and the metric being used to calculate it before the table/list of operators (this is not shown in the screenshot and is left as implementation detail). It will have a Refresh button as shown in the screenshot similar to the Checkpoints tab.

A Note on Using Feature Flag / Config for Enabling the Proposed Changes

The proposed changes should only be improving the user experience without any unwanted side-effect. Therefore this proposal does not suggest using a feature flag or config to enable this new feature. However, I don't consider this a major contentious point for the FLIP and am happy to revisit this before/during the implementation phase (e.g. after testing impact on UI).

Compatibility, Deprecation, and Migration Plan

...

• Test case for the metric where the result is undefined i.e. idle operator. We will show 0 as the data skew score on the UI
• Boundary tests: Ensure result can't go beyond 100% or below 0%
• Define test scenarios for 5%, 50%, 99% data skew score and automate the tests

...