Page History

...

• ResourceSpec (See Resource Requirements)
• Introduce new configuration "taskmanager.defaultSlotResourceFraction", while deprecate but stay compatible with the configuration "taskmanager.numberOfTaskSlots". (See Unknown Resource Requirements)
• RestAPI / WebUI (Need to adapt the RestAPI and WebUI to the dynamic slot model.)

...

• CPU cores
• Unmanaged on-heap memory (Task Heap Memory in FLIP-49)
• Unmanaged off-heap memory (Task Off-Heap Memory in FLIP-49)
• Managed on-heap memory (On-Heap Managed Memory in FLIP-49)
• Managed off-heap memory (Off-Heap Managed Memory in FLIP-49)
• Network memory (Network Memory in FLIP-49)
• Extended resources (GPU, FPGA, etc.)

Operators declare their resource requirements in ResourceSpec. A ResourceSpec should contain all the above resource dimensions, except for network memory, which can be derived in at runtime from the execution graph topology. Among the dimensions, CPU cores and unmanaged on-heap memory (which all operators must consume) are required as long as the operator declares a ResourceSpec, while other dimensions are optional and will be set to 0 by default if not explicitly specified.

If operators do not specify any ResourceSpec, their resource requirements are by default UNKNOWN, which will leave it to the runtime to decide how many resources they can consume.

For the first version, we do not support mixing operators with specified / unknown resource requirements in the same job. Either all or none of the operators of the same job should specify their resource requirements. StreamGraphGenerator should check this and throw an error when mixing of specified / unknown resource requirements is detected, during the compiling compilation stage.

Managed Memory Allocation

...