Status

Current state: Under Discussion Accepted

Discussion thread: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-49-Unified-Memory-Configuration-for-TaskExecutors-td31436.html

JIRA: https://jira.apache.org/jira/browse/FLINK-13980

Released:

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

• JVM heap memory
• Includes Framework Heap Memory, Task Heap Memory, and On-Heap Managed Memory
• Explicitly set both  -Xmx and -Xms to this value
• JVM direct memory
  • Includes Task Off-heap Memory , and Shuffle Memory and JVM Overhead
  • Explicitly set -XX:MaxDirectMemorySize to this value
  • For Off-heap Managed Memory, we always allocate memory with Unsafe.allocateMemory(), which will not be limited by this parameter.
• JVM metaspace
• Set -XX:MaxMetaspaceSize to configured JVM Metaspace

...

• We need to update existing and add new integration tests dedicated to validate the new memory configuration behaviors.
• It is also expected that other regular integration and end-to-end tests should fail if this is broken.

Limitations

• The proposed design uses Unsafe.allocateMemory() for allocating managed memory, which is no longer supported Java 12. We need to look for alternative solutions in the future.

Follow Ups

• This FLIP requires very good documentation to help users understand how to properly configure Flink processes and which knobs to turn in which cases.
• It would be good to expose configured memory pool sizes in the web UI, so that users see immediately what amount of memory TMs assume to use for what purpose.

Rejected Alternatives

Regarding JVM direct memory, we have the following alternative.

...