Page History

...

(taskmanager.memory.task.offheap) 

(default 0b)

...

Shuffle Memory

Off-heap memory for network buffersshuffling.

(taskmanager.memory.shuffle.[min/max/fraction]) or (taskmanager.network.memory.[min/max/fraction])

(default min=64mb, max=1gb, fraction=0.1)

...

This includes Framework Heap Memory, Task Heap Memory, Task Off-Heap Memory, Network Memory Shuffle Memory, and Managed Memory.

This excludes JVM Metaspace and JVM Overhead.

...

• JVM heap memory
• Includes Framework Heap Memory, Task Heap Memory, and On-Heap Managed Memory
• Explicitly set both  -Xmx and -Xms to this value
• JVM direct memory
  • Includes Task Off-heap Memory, Network Memory Shuffle Memory and JVM Overhead
  • Explicitly set -XX:MaxDirectMemorySize to this value
  • For Off-heap Managed Memory, we always allocate memory with Unsafe.allocateMemory(), which will not be limited by this parameter.
• JVM metaspace
• Set -XX:MaxMetaspaceSize to configured JVM Metaspace

...

• If both Task Heap Memory and Managed Memory are configured, we use these to derive Total Flink Memory
• If Network MemoryShuffle Memory is configured explicitly, we use that value
• Otherwise, we compute it such that it makes up the configured fraction of the final Total Flink Memory (see getAbsoluteOrInverseFraction())
• If Total Flink Memory is configured, but not Task Heap Memory and Managed Memory, then we deriveNetwork  Shuffle Memory and Managed Memory, and leave the rest (excluding Framework Heap Memory and Task Off-Heap Memory) as Task Heap Memory.
• IfNetwork  Shuffle Memory is configured explicitly, we use that value
• Otherwise we compute it such that it makes up the configured fraction of the Total Flink Memory (see getAbsoluteOrFraction())
• If Managed Memory is configured explicitly, we use that value
• Otherwise we compute it such that it makes up the configured fraction of the Total Flink Memory (see getAbsoluteOrFraction())
• If only the Total Process Memory is configured, we derive the Total Flink Memory in the following way
• We get (or compute relative) and subtract the JVM Overhead from Total Process Memory (see getAbsoluteOrFraction())
• We subtract JVM Metaspace from the remaining
• We leave the rest as Total Flink Memory

...