Page History

...

...

• If the JV of the SEV doesn’t contain co-location constraint
  • If the parallelism of JV is equals to numberOfSlots
    • Traverse all SEVs of JV, assign the SEVs[subtask_index] to the ESSGs[subtask_index]This strategy ensures that SEVs with the same index can be assigned to the same ESSG. 
    • It also ensures that co-located subtasks  will be in one ESSG, given that co-located job vertices share the same parallelism and will always be in the same SSG. 
    • In the case of forward edges, remote data exchange of such JVs can be reduced, all subtasks with forward shuffle are still in the same Slot, and they are local data exchange. 

• • If the parallelism of JV is less than the numberOfSlots in the current SSG.
    • Get  ESSGs[++eSSGIndex]  as ESSG.
    • Add the SEV into the target ESSG.

...

• Expose two options to the user
  • slot.sharing-strategy and cluster.evenly-spread-out-slots
  • After discussing at mail list, we unify these 2 option into a option: taskmanager.load-balance.mode.
  • Related discussions at here:
    • https://lists.apache.org/thread/fsssvzodpkzw87nt2kx1slsdzkk3m30t
    • https://lists.apache.org/thread/h5zph52ovwxfhsj4rgm461jvlcowzody
    • https://lists.apache.org/thread/lkxvxp9mvb3wqf9114dz7osx5nr1vlpn
• Change the strategy of slot.request.max-interval
  • The old strategy is: when `current time > arrival time of the last slotRequest + slot.request.max-interval` and `the number of available slots cached exceeds or equals the pending slots`, the slots selection and allocation can be carried out.
  • The latest strategy is:
    • For DeclarativeSlotPool#increaseResourceRequirementsBy, When `current time > arrival time of the last slotRequest + slot.request.max-interval`, then the DeclarativeSlotPool declareResourceRequirement to the RM.
    • For DeclarativeSlotPool#decreaseResourceRequirementsBy, When `current time > arrival time of the last slotRequest + slot.request.max-interval`, then the SlotPool DeclarativeSlotPool declareResourceRequirement to  to the RM.
    • For DeclarativeSlotPoolBridge, When `the number of available slots cached exceeds or equals the pending slots`, the slots selection and allocation can be carried out.
  • This new strategy not only implements the functionality of the old strategy, but also reduces the number of rpcs from SlotPool to ResourceManager.

...