...

Task executor memory configuration options. (See Memory Pools and Configuration Keys)A summary of backwards compatibility  

Proposed Changes

Unifying Managed Memory for Batch and Streaming

...

This excludes JVM Metaspace and JVM Overhead.

(taskmanager.memory.total-flink.size)

Total Process Memory

...

This includes Total Flink Memory, and JVM Metaspace and JVM Overhead.

(taskmanager.memory.total-process.size)

JVM Parameters

...

• Introduce new configuration options
• Introduce TaskExecutorSpecificsdata structures and utilities.
  • Data structure to store memory / pool sizes of task executor
  • Utility for calculating memory / pool sizes from configuration
  • Utility for generating dynamic configurations
  • Utility for generating JVM parameters

...

Step 3. Launch task executor with new memory calculation logics

• Invoke TaskExecutorSpecifics data structures and utilities introduced in Step 2 to generate JVM parameters and dynamic configurations for launching new task executors.
  • In startup scripts
  • In resource managers
• Task executor uses TaskExecutorSpecifics to uses data structures and utilities introduced in Step 2 to set memory pool sizes and slot resource profiles.
  • MemoryManager
  • ShuffleEnvironment
  • TaskSlotTable

...

Compatibility, Deprecation, and Migration Plan

This FLIP changes how users configure cluster resources, which in some cases may require re-configuring of cluster if migrated from prior versions.

Deprecated configuration keys are as follows:

Test Plan

• We need to update existing and add new integration tests dedicated to validate the new memory configuration behaviors.
• It is also expected that other regular integration and end-to-end tests should fail if this is broken.

...