...

Public Interfaces

Proposed Changes

We will split this JIRA into two tasks.

The first task: Ignoring max buffers per channel when allocate buffer

The LocalBufferPool will be unavailable when the maxBuffersPerChannel is reached for this channel or availableMemorySegments.isEmpty.

If we request a memory segment from LocalBufferPool and the maxBuffersPerChannel is reached for this channel, we just ignore that and continue to allocate buffer while availableMemorySegments isn't empty in LocalBufferPool.

The second task: Adding the overdraft buffer

Adding the new configuration : taskmanager.network.memory.max-overdraft-buffers-per-gate, the default value is 5~10?

The LocalBufferPool will be unavailable when the maxBuffersPerChannel is reached for this channel or availableMemorySegments.isEmpty or numberOfRequestedOverdraftMemorySegments > 0.

If we request a memory segment, we try to allocate buffer from availableMemorySegments. If availableMemorySegments.isEmpty numberOfRequestedOverdraftMemorySegments < maxOverdraftBuffersPerGate, we will request the overdraftBuffer from NetworkBufferPool.

For return memory segment, we will return the overdraft first if numberOfRequestedOverdraftMemorySegments > 0.

Compatibility, Deprecation, and Migration Plan

...