...

        Math.max(min, Math.min(relative, max))

    }

}

Implementation Steps

Step 1. Introduce a switch for enabling the new task executor memory configurations

Introduce a temporal config option as a switch between the current / new task executor memory configuration code paths. This allows us to implement and test the new code paths without affect the existing code paths and behaviors.

Step 2. Implement memory calculation logics

• Introduce new configuration options
• Introduce TaskExecutorSpecifics
  • Data structure to store memory / pool sizes of task executor
  • Utility for calculating memory / pool sizes from configuration
  • Utility for converting between TaskExecutorSpecifics and environment variables
  • Utility for generating JVM parameters

This step should not introduce any behavior changes.

Step 3. Launch task executor with new memory calculation logics

• Invoke TaskExecutorSpecifics to generate JVM parameters and environment variables for launching new task executors.
  • In startup scripts
  • In resource managers
• Task executor uses TaskExecutorSpecifics (converted from environment variables) to set memory pool sizes and slot resource profiles.
  • MemoryManager
  • ShuffleEnvironment
  • TaskSlotTable

Implement this step as separate code paths only for the new mode.

Step 4. Separate on-heap and off-heap managed memory pools

• Update MemoryManager to have two separated pools.
• Extend MemoryManager interfaces to specify which pool to allocate memory from.

Implement this step in common code paths for the legacy / new mode. For the legacy mode, depending to the configured memory type, we can set one of the two pools to the managed memory size and always allocate from this pool, leaving the other pool empty.

Step 5. Support reserving managed memory.

• Disable pre-allocation.
• Extend MemoryManager to support memory reserving.

Implement this step in common code paths for the legacy / new mode. This introduces behavior changes to legacy mode by disable pre-allocation, which only affects performance without breaking any functionality.

Step 6. Use native memory for network and managed memory.

• Allocate memory with Unsafe.allocateMemory
  • MemoryManager
  • ShuffleEnvironment

Implement this issue in common code paths for the legacy / new mode. This should only affect the GC behavior.

Step 7. Clean-up of legacy mode.

• Fix / update / remove test cases for legacy mode
• Deprecate / remove legacy config options.
• Remove legacy code paths
• Remove the switch for legacy / new mode.

Compatibility, Deprecation, and Migration Plan

...