THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Status
...
| Page properties | ||||
|---|---|---|---|---|
|
...
...
|
...
|
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
We will split this JIRA into two 3 tasks.
The first task:
...
Ignore max buffers per channel when allocate buffer
| Jira | ||||||
|---|---|---|---|---|---|---|
|
The LocalBufferPool will be unavailable when the maxBuffersPerChannel is reached for this channel or availableMemorySegments.isEmpty.
...
The second task: Adding the overdraft buffer
| Jira | ||||||
|---|---|---|---|---|---|---|
|
Adding the new configuration : taskmanager.network.memory.max-overdraft-buffers-per-gate, the default value is 5~10?5.
The LocalBufferPool will be unavailable when the maxBuffersPerChannel is reached for this channel or availableMemorySegments.isEmpty or numberOfRequestedOverdraftMemorySegments > 0.
...
For return memory segment, we will return the overdraft first if numberOfRequestedOverdraftMemorySegments > 0.
The third task: Compatible with LegacySource
| Jira | ||||||
|---|---|---|---|---|---|---|
|
Since LegacySource does not have checkAvailable, LegacySource will use all overdraft buffers by default, this is not what we expected.
So we'll set overdraft=0 for the SourceStreamTask.
Compatibility, Deprecation, and Migration Plan
...
- Test for apply for overdraft buffer when overdraft buffer is sufficient
- Test for apply for overdraft buffer when overdraft buffer is insufficient
- Checkpoint Duration Benchmark for enable overdraft buffer
Rejected Alternatives
After discussing, we decided to use overdraft-buffer instead of reserve-buffer. For details, please refer to the mail list.