THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
| Config Option | Description | ||
|---|---|---|---|
| taskmanager.network.sort-merge-blocking-shuffle.max-filesbuffers-per-partition | The maximum Maximum number of files can be produced by each sort-merge blocking partition, files over this threshold will be merged. | taskmanager.network.sort-merge-blocking-shuffle.buffers-per-partition | Number of network buffers required for each required network buffers can be used per sort-merge blocking result partition. Larger value can reduce the number of shuffle files and bring better performance. |
| taskmanager.network.sort-merge-blocking-shuffle.min-parallelismenabled | Whether to use For small parallelism, hash-based blocking shuffle will be used and for large parallelism, sort-merge based blocking shuffle will be usedor not. |
A fixed number of network buffers per result partition makes the memory consumption decoupled with parallelism which is more friendly for large scale batch jobs.
...