Page History

...

• Add a new config option.
  • Key
    • taskmanager.network.memory.read-buffer.required-per-gate.max
  • Default
    • (none).
  • Type
    • Integer.
  • Description
    • The maximum number of network read buffers that are required by an input gate. (An input gate is responsible for reading data from all subtasks of an upstream task.) The number of buffers needed by an input gate is dynamically calculated in runtime, depending on various factors (e.g., the parallelism of the upstream task). Among the calculated number of needed buffers, the part below this configured value is required, while the excess part, if any, is optional. A task will fail if the required buffers cannot be obtained in runtime. A task will not fail due to not obtaining optional buffers, but may suffer a performance reduction. If not explicitly configured, the default value is Integer.MAX_VALUE for streaming workloads, and 1000 for batch workloads.
  • In the first version, this new config will be annotated as @Experimental. We will revisit and try to remove the annotation in the next version.
• Deprecated 2 existing config options
  • taskmanager.network.memory.buffers-per-channel
  • taskmanager.network.memory.floating-buffers-per-gate
  • We will not annotate these 2 configs as @Deprecated until the @Experimental annotation for taskmanager.network.memory.read-buffer.required-per-gate.max is removed.
• Modify the default value of taskmanager.memory.network.max from 1g to MemorySize.MAX_VALUE.

...