Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

However, for other types of jobs, such as those running on YARN or in standalone mode, the default behavior involves persisting scaling information in memory itself via introducing new implementation HeapedAutoScalerStateStore . This means that the state information is stored in memory and can be lost if the autoscaler restarts. It It's important to note that, in the future, there is the possibility to pesist in RDBMS or any persistent storage. It can be new implementation such as JdbcAutoScalerStateStore  etc  to ensure persistent storage of the state.

...

As discussed with Gyula Fora  and Samrat Deb ,  yarn autoscaler implementation will be out of scope for this FLIP. we We will develop a generic autoscaler based on the rescale API (FLIP-291). This generic autoscaler will not have knowledge of specific jobs, and users will have the flexibility to pass the JobAutoScalerContext when utilizing the generic autoscaler. Communication with Flink jobs can be achieved through the RestClusterClient. 

  • The generic ScalingRealizer based on the rescale API (FLIP-291).
  • The generic EventHandler based on the logger.
  • The generic StateStore based on the Heap. This means that the state information is stored in memory and can be lost if the autoscaler restarts.

4. Proposed Changes

4.1 Ensure new autoscaler module keeps the generic auto scaling strategy.

...