Status

...

...

...

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

Public Interfaces

Proposed Changes

We will split this JIRA into 3 tasks.

The first task: Ignore max buffers per channel when allocate buffer

The LocalBufferPool will be unavailable when the maxBuffersPerChannel is reached for this channel or availableMemorySegments.isEmpty.

If we request a memory segment from LocalBufferPool and the maxBuffersPerChannel is reached for this channel, we just ignore that and continue to allocate buffer while availableMemorySegments isn't empty in LocalBufferPool.

The second task: Adding the overdraft buffer

Adding the new configuration : taskmanager.network.memory.max-overdraft-buffers-per-gate, the default value is 5.

The LocalBufferPool will be unavailable when the maxBuffersPerChannel is reached for this channel or availableMemorySegments.isEmpty or numberOfRequestedOverdraftMemorySegments > 0.

If we request a memory segment, we try to allocate buffer from availableMemorySegments. If availableMemorySegments.isEmpty numberOfRequestedOverdraftMemorySegments < maxOverdraftBuffersPerGate, we will request the overdraftBuffer from NetworkBufferPool.

For return memory segment, we will return the overdraft first if numberOfRequestedOverdraftMemorySegments > 0.

The third task: Compatible with LegacySource

Since LegacySource does not have checkAvailable, LegacySource will use all overdraft buffers by default, this is not what we expected.

So we'll set overdraft=0 for the SourceStreamTask.

Compatibility, Deprecation, and Migration Plan

...

• Test for apply for overdraft buffer when overdraft buffer is sufficient
• Test for apply for overdraft buffer when overdraft buffer is insufficient
• Checkpoint Duration Benchmark for enable overdraft buffer 

Rejected Alternatives

After discussing, we decided to use overdraft-buffer instead of reserve-buffer. For details, please refer to the mail list.