Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • Flink should rescale immediately only if last rescale was done more than scaling-interval.min ago.
  • Otherwise it should schedule a rescale at (now + scaling-interval.min) point in time. It is equivalent to resetting the cooldown period when new slots arrive during a cooldown period. Indeed, we decided to lower the scaling-interval.min default value to be more reactive (cf compatibility part), resetting the period allows to protect against too frequent rescales. 
  • The rescale is done like this: 
    • if minimum scaling requirements are met (AdaptiveScheduler#shouldRescale), the job is restarted with new parallelism (as before)
    • if minimum scaling requirements are not met but last rescale was done more than scaling-interval.max ago, a rescale is forced.

...

So this FLIP proposes setting defaults values to jobmanager.adaptive-scheduler.scaling-interval.min = 30s and no jobmanager.adaptive-scheduler.scaling-interval.max (force scaling disabled). Indeed, by default we prefer to favor lower numbers scaling-interval.min (for smooth rescale experiencemore reactive rescaling) and consider higher numbers as exceptions set by let the users when increase the value when they have high workloads.

...